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during  a  time  of  war  and  change.  General  Bolton  was  an  admired  mentor  to  many 
pilots  and  military  acquisition  professionals.  We  knew  him  best  as  a  consumer  and 
reviewer  of  RAND  research,  including  the  work  documented  in  this  report.  We  often 
sought  his  advice  and  counsel  and  will  miss  his  wisdom  and  collegiality. 


Preface 


As  a  result  of  continuing  concern  about  large  cost  overruns  in  a  broad  range  of  major 
defense  programs,  Congress  enacted  new  statutory  provisions  extending  the  ambit  of 
the  existing  Nunn-McCurdy  Act.  In  response  to  congressional  direction,  the  Depart¬ 
ment  of  Defense  established  the  Performance  Assessments  and  Root  Cause  Analyses 
(PARCA)  office  to  coordinate  the  responses  to  such  breaches.  Since  then,  PARCA  has 
engaged  the  RAND  Corporation  to  conduct  multiple  studies  on  the  root  causes  of 
Nunn-McCurdy  breaches  or  other  large  cost  increases  in  major  defense  acquisition 
programs.  In  addition  to  reports  on  major  defense  acquisition  programs,  RAND,  at 
the  request  of  the  sponsor,  has  researched  topics  related  to  the  management  of  defense 
acquisition.  This  research  includes  such  topics  as  program  manager  tenure,  oversight  of 
Acquisition  Category  II  programs,  framing  assumptions,  and  programs  with  multiple 
Nunn-McCurdy  breaches. 

This  report  provides  the  results  of  two  analyses  requested  by  the  PARCA  office. 
One  is  the  analysis  performed  by  RAND  on  the  most  recent  root  cause  analysis:  the 
Joint  Precision  Approach  and  Landing  System  Increment  1A.  The  second  analysis 
responds  to  an  interest  expressed  by  PARCA  in  developing  a  clear  and  replicable  way 
to  assess  and  summarize  the  overall  performance  of  an  acquisition  portfolio,  both  at  a 
single  point  in  time  and  over  several  years.  PARCA  asked  RAND  for  help  in  develop¬ 
ing  and  validating  a  methodology  that  could  be  used  by  analysts  within  the  defense 
acquisition  community  to  conduct  this  type  of  portfolio  assessment.  The  report  docu¬ 
ments  the  methodology  RAND  researchers  developed  in  support  of  this  request  and 
the  work  they  conducted  to  refine,  validate,  and  demonstrate  the  methodology.  Note 
that  these  two  analyses  are  not  analytically  connected  but  rather  represent  separate 
responses  to  two  requests  from  PARCA. 

This  research  was  sponsored  by  the  Office  of  the  Secretary  of  Defense  (OSD) 
PARCA  office  and  conducted  within  the  Acquisition  and  Technology  Policy  Center 
of  the  RAND  National  Defense  Research  Institute,  a  federally  funded  research  and 
development  center  sponsored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff, 
the  Unified  Combatant  Commands,  the  Navy,  the  Marine  Corps,  the  defense  agen¬ 
cies,  and  the  defense  Intelligence  Community. 


vi  Preface 


For  more  information  on  the  RAND  Acquisition  and  Technology  Policy  Center, 
see  www.rand.org/nsrd/ndri/centers/atp  or  contact  the  director  (contact  information  is 
provided  on  the  web  page). 
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Summary 


This  report  provides  the  results  of  two  analyses  requested  by  the  Performance  Assess¬ 
ments  and  Root  Cause  Analyses  (PARCA)  office.  One  is  a  root  cause  analysis  of  the 
Nunn-McCurdy  breach  by  the  Joint  Precision  Approach  and  Landing  System  (JPALS) 
Increment  1A  program.  The  second  analysis  responds  to  PARCA’s  request  for  help  in 
developing  a  clear  and  replicable  way  to  assess  and  summarize  the  overall  performance 
of  an  acquisition  portfolio,  both  at  a  single  point  in  time  and  over  several  years. 


The  JPALS  Increment  1A  Program 

Background  and  Purpose  of  JPALS  Program 

In  the  early  1990s,  the  Department  of  Defense  (DoD)  began  to  pursue  a  precision 
landing  system  that  would  help  in  adverse  weather  conditions.  U.S.  operations  in 
Bosnia,  which  were  run  from  a  relatively  austere  airfield  with  limited  air  traffic  control 
capabilities  and  encountered  recurring  delays,  demonstrated  this  need.  The  result  was 
the  Global  Positioning  System  (GPS)-based  JPALS  acquisition  program.  This  was  an 
ambitious  program,  not  only  because  it  involved  all  three  military  services  but  also 
because  it  involved  a  civilian  agency,  the  Federal  Aviation  Administration  (FAA),  and 
had  international  implications  because  many  foreign  airfields  used  the  Instrument 
Landing  System  (ILS),  which  at  the  time  was  also  the  Air  Force’s  primary  landing 
system. 

The  program  had  three  goals: 

•  Give  U.S.  forces  the  ability  to  land  on  an  aircraft  carrier’s  deck  or  a  primitive  air¬ 
strip  in  bad  weather  when  visibility  is  low. 

•  Provide  accurate  information,  even  in  the  face  of  enemy  attempts  to  jam  the  land¬ 
ing  signal,  falsify  landing  data,  or  destroy  ground  devices. 

•  Work  with  all  Air  Force,  Navy,  and  Army  platforms  as  well  as  commercial  and 
allied  aircraft. 

Initially,  the  Air  Force  led  the  project  and  considered  a  variety  of  systems.  JPALS 
was  expected  to  cost  between  $1.8  billion  and  $3.5  billion,  which  would  be  shared 
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between  the  Army,  Navy,  and  Air  Force.* 1  The  Air  Force  led  a  joint  military  and  FAA 
working  group  to  study  possible  precision  approach  and  landing  systems,  eventually 
concluding  that  no  single  electronic  package  could  meet  all  the  needs  identified  by  the 
JPALS  team.2  It  was  finally  determined  that  not  one  but  two  hybrid  solutions  would 
be  required  because  DoD  opposed  the  idea  of  equal  military  and  civilian  access  to  the 
GPS  signal. 

A  number  of  important  changes  occurred  during  the  early  years  of  the  program. 
These  included,  among  others,  decreased  budget,  technological  advances,  assumptions 
about  what  would  be  the  standard  precision  approach  and  landing  technology,  and 
doubts  about  what  technological  course  the  FAA  would  follow.  Eventually,  the  Navy 
assumed  the  lead  on  the  program  because  its  requirements  were  addressed  first  in  the 
acquisition  program  defined  at  Milestone  B  in  2008 — JPALS  Inc.  1A.  In  January 
2014,  the  Navy  informed  the  Under  Secretary  of  Defense  for  Acquisition,  Technology 
and  Logistics  (USD  [AT&L])  that  the  JPALS  Inc.  1A  acquisition  program  was  going 
to  have  a  critical  Nunn-McCurdy  unit  cost  breach.3 

The  purpose  of  the  JPALS  analysis  is  to  identify  the  root  cause  (i.e.,  the  under¬ 
lying  cause  or  causes  of  shortcomings  in  cost,  schedule,  or  performance  of  the  pro¬ 
gram)  that  triggered  the  Nunn-McCurdy  breach.4’5 6 7 8 


1  “JPALS  Needs  Additional  $12  Million  for  DEM/VAL,  According  to  PBD,”  Inside  the  Air  Force,  December  20, 

1996. 

2  “JPALS  on  Tight  Schedule  for  Improving  Aircraft  Approach  and  Landing,”  Inside  the  Air  Force,  March  14, 

1997. 

3  CAPT  Darrell  D.  Lack,  U.S.  Navy,  “Program  Deviation  Report  Addendum  for  the  Joint  Precision  Approach 
and  Landing  System  Increment  1A  Program,”  memorandum  for  the  Under  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics),  Department  of  the  Navy,  Program  Executive  Office,  Tactical  Aircraft  Programs, 
January  28,  2014. 

4  The  WSARA  legislation  (Public  Law  1 1 1-23,  Weapon  Systems  Reform  Act  of 2009,  May  22,  2009)  defines  root 
cause  analysis  as  the  following: 

(d)  ROOT  CAUSE  ANALYSES. — For  purposes  of  this  section  and  section  2433a  of  title  10,  United  States  Code  (as 
so  added),  a  root  cause  analysis  with  respect  to  a  major  defense  acquisition  program  is  an  assessment  of  the  underly¬ 
ing  cause  or  causes  of  shortcomings  in  cost,  schedule,  or  performance  of  the  program,  including  the  role,  if  any,  of — 

(1)  unrealistic  performance  expectations; 

(2)  unrealistic  baseline  estimates  for  cost  or  schedule; 

(3)  immature  technologies  or  excessive  manufacturing  or  integration  risk; 

(4)  unanticipated  design,  engineering,  manufacturing,  or  technology  integration  issues  arising  during  program 
performance; 

(5)  changes  in  procurement  quantities; 

(6)  inadequate  program  funding  or  funding  instability; 

(7)  poor  performance  by  government  or  contractor  personnel  responsible  for  program  management;  or 

(8)  any  other  matters. 

5  The  National  Aeronautics  and  Space  Administration  (NASA)  also  has  defined  the  concept  of  root  cause  analy¬ 
sis  given  that  it  uses  it  for  investigations  into  problems  with  its  space  vehicles,  etc.  In  2003,  NASA  defined  root 
cause  analysis  as  “a  structured  evaluation  method  that  identifies  the  root  causes  of  an  undesired  outcome  and  the 
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Root  Cause  of  the  JPALS  Nunn-McCurdy  Breach 

Under  current  law,  if  a  program  exceeds  certain  cost  thresholds,  it  is  designated  as 
having  a  Nunn-McCurdy  breach.  A  breach  can  occur  if  either  the  average  procure¬ 
ment  unit  cost  (APUC)  and  program  acquisition  unit  cost  (PAUC)  exceeds  a  stipulated 
threshold.  Depending  on  how  much  the  program  exceeds  the  threshold,  the  breach 
can  be  classified  as  either  significant  or  critical.  These  limits  appear  in  Table  S.l. 

Both  the  APUC  and  the  PAUC  for  the  JPALS  Inc.  1A  program  exceeded  critical 
thresholds  against  both  the  original  baseline  and  the  current  baseline  triggering  the 
Nunn-McCurdy  process.* * 6  Table  S.2,  the  format  of  which  has  been  dubbed  the  “speed¬ 
ing  ticket”  for  an  acquisition  program  that  breaches  a  threshold,  shows  the  results  in 
columns  four  and  five  (labeled  Baseline  Breached  and  percentage).  The  APUC  exceeded 
both  baselines  by  nearly  129  percent,  and  the  PAUC  was  approximately  104  percent 
over  both  baselines.  The  reasons  for  those  breaches  appear  in  the  tenth  column  (Cause 
in  Source).  The  cause  of  the  breach  was  the  reduction  in  the  number  of  systems,  from 
25  to  only  17.  This  reduction  made  each  system  substantially  more  expensive. 

Our  analysis  found  that  the  root  cause  of  this  Nunn-McCurdy  breach  was  the 
FAA’s  decision  to  continue  using  ILS  instead  of  phasing  it  out  and  phasing  in  GPS- 
based  precision  approach  and  landing  technology.  This  planning  diverged  from  DoD’s 


Table  S.l 

Nunn-McCurdy  Breach  Limits 


Level 

Unit  Cost 

Baseline 

Threshold 

Significant 

PAUC 

Current 

>15% 

APUC 

Current 

>15% 

PAUC 

Original 

>30% 

APUC 

Original 

>30% 

Critical 

PAUC 

Current 

>25% 

APUC 

Current 

>25% 

PAUC 

Original 

>50% 

APUC 

Original 

>50% 

actions  adequate  to  prevent  recurrence”  (Faith  Chandler,  NASA  Root  Cause  Analysis  Supplemental  Training  Mate¬ 

rial,  Part  I:  NASA  RCA,  National  Aeronautics  and  Space  Administration,  March  25,  2010,  Foreword). 

6  APUC  is  calculated  by  dividing  total  procurement  cost  by  the  number  of  articles  to  be  procured,  and  PAUC 
is  calculated  by  dividing  program  acquisition  cost  by  the  program  acquisition  quantity.  The  Nunn-McCurdy 
legislation  established  thresholds  that,  when  crossed,  require  Congress  to  be  notified.  A  breach  can  be  classified 
as  either  significant  or  critical,  depending  on  how  much  the  cost  exceeds  the  original  or  current  baseline  cost  esti¬ 
mate  of  the  program.  For  example,  a  PAUC  that  exceeds  the  current  baseline  estimate  by  25  percent  would  be  a 
critical  breach. 


Table  S.2 

JPALS  Inc.  1A  Speeding  Ticket 


Cost  Growth  Threshold  Breaches 

Current  - 

Estimate  (Source,  Current 

Baseline  Dec  31, 2013,  Quantity 

Unit  Cost  (FY  SAR)  FY  2008  Baseline3  Baseline  (Dec  2013  Cause  in  Explanation 


Program 

$  millions) 

$  millions 

Breached 

Percentage 

Amount 

Level 

Quantity 

SAR) 

Source 

in  Source 

JPALS  Inc. 

APUC 

APUC 

Over  current 

APUC 

+$10,466 

Critical 

25 

17 

Reduction  in  total 

As  a  result  of 

1 A 

$8,116 

$18,582 

baseline 

+128.96% 

FY  2008  $M 

planned  procurement 

the  Navy's  PALC 

(Dec  2008 

(Dec  2013 

quantities  resulted  in  a 

Roadmap,  it  was 

APB) 

SAR) 

critical  Nunn-McCurdy 

determined  that 

unit  cost  breach  to  the 

previously  required 

PAUC 

PAUC 

PAUC 

+$27,146 

Critical 

37 

27 

current/original  APB 

shore-based  training 

$26,032 

$53,178 

+104.28% 

FY  2008  $M 

systems  would  be 

(Dec  2008 

eliminated 

APB) 

APUC 

APUC 

Over  original 

APUC 

+$10,466 

Critical 

25 

17 

$8,116 

$18,582 

baseline 

+128.96% 

FY  2008  $M 

(Dec  2008 

(Dec  2013 

APB) 

SAR) 

PAUC 

PAUC 

PAUC 

+$27,146 

Critical 

37 

27 

$26,032 

$53,178 

+014.28% 

FY  2008  $M 

(Dec  2008 
APB) 


SOURCE:  Department  of  Defense,  Joint  Precision  Approach  and  Landing  System  Increment  1A  (JPALS  Inc.  1A),  Selected  Acquisition  Report, 
December  2013b. 

NOTE:  The  numbers  in  red  indicate  the  "speeding  ticket"  triggering  root  cause  analysis  by  PARCA. 

3  For  the  JPALS  Inc.  1A  program,  both  the  original  and  current  estimates  are  the  same.  The  program  had  a  Milestone  B  in  2008  in  which  both 
the  current  and  original  estimates  started  as  the  same  estimate  based  on  the  APB  at  Milestone  B.  After  Milestone  B,  there  was  little  cost  growth 
or  other  changes  to  the  program  that  would  have  warranted  a  change  in  the  current  baseline;  therefore,  the  current  and  original  baselines 
remained  the  same  until  the  Nunn-McCurdy  breach. 
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plan  to  move  toward  GPS -based  technology.  As  a  result,  the  Army  and  the  Air  Force 
reassessed  their  precision  approach  and  landing  capability  (PALC)  plans  and  eventu¬ 
ally  pulled  out  of  the  program.  The  Navy  then  had  to  consider  the  consequences  of  the 
Air  Force,  Army,  and  FAA  continuing  to  rely  on  ILS.  More  specifically,  the  Navy’s  pro¬ 
posed  restructuring  (and  funding  consolidation),  which  was  motivated  by  the  findings 
of  the  Navy’s  PALC  Roadmap  study,  was  initiated  by  a  series  of  “fact-of-life”  changes 
to  the  assumptions  underpinning  the  original  JPALS  seven-increment  plan.  Figure  S.l 
provides  the  series  and  timing  of  the  events  discussed  above. 

In  addition  to  the  ultimate  decision  by  the  Navy  to  restructure  the  JPALS 
program,  the  baseline  program — Inc.  1A — incurred  some  modest  cost  and  schedule 
growth,  independent  of  the  factors  affecting  the  Navy’s  decision  to  restructure  the 

Figure  S.l 

Root  Cause  of  the  JPALS  Inc.  1A  Nunn-McCurdy  Breach 


aAccording  to  a  November  2012  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ)  memo,  because  of 
recent  affordability  assessment,  ongoing  service  life  extension  on  legacy  systems,  and  a  change  in 
operational  need.  The  Army  did  not  release  an  official  memo,  but  zeroed  out  funding  in  April  2013 
budget  documents  as  did  the  Air  Force.  The  Army  has  a  standing  unfunded  requirement. 
bA  2005  Analysis  of  Alternatives  (AoA)  identified  gaps:  operational  interoperability  with  FAA,  DoD,  and 
international  civil  aviation  organizations;  supportability  in  all  four  operating  environments;  and  an 
emerging  gap  in  sea-based  operations  beginning  in  2013  (CVN21,  Joint  Strike  Fighter,  Joint  Unmanned 
Combat  Air  System). 

cAccording  to  multiple  FAA  official  planning  documents  (2001-2012),  the  phasedown  of  the  standard 
precision  and  landing  system,  ILS,  in  favor  of  GPS-based  precision  approach  and  landing  (PAL)  technology 
did  not  begin  as  was  planned  in  2001 .  After  2008,  the  transition  remained  unclear. 
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program.  The  JPALS  Inc.  1A  program  of  record  required  shipboard  integrated  testing 
and  operational  assessment  (OA).  The  testing  was  planned  to  occur  on  the  CVN-77. 
According  to  the  Program  Deviation  Report  from  July  2012,  several  shifts  from  2009 
to  2012  to  CVN-77  installation  availability  occurred,  delaying  OA  testing  and  conse¬ 
quently  causing  a  slip  in  Milestone  C  from  May  2013  to  November  2013,  which  was 
an  APB  schedule  breach.  This  shift  also  resulted  in  a  decrease  of  one  research,  devel¬ 
opment,  test,  and  evaluation  (RDT&E)  unit  (from  12  to  11)  and  an  increase  of  one 
procurement  unit  for  a  new  total  of  27  procurement  units.  The  revised  procurement 
quantity  was  needed  to  satisfy  a  new  production  schedule,  which  was  extended  by 
three  years.  The  increase  in  procurement  units  and  fixed  costs  caused  an  APB  procure¬ 
ment  cost  breach.7 

Using  data  on  the  program  components  that  drove  costs  from  each  of  the  annual 
Selected  Acquisition  Reports  (SARs),  we  found  that  RDT&E  grew  an  additional 
$334.6  million  because  of  the  restructuring  of  the  program  leading  to  increased  scope 
from  Increments  3  and  4.  Increments  3  and  4  were  not  fully  defined,  so  the  incor¬ 
poration  of  these  dollar  values  into  Increment  1A  introduced  new  uncertainty.  For 
instance,  the  period  of  development  was  extended  in  the  restructure  at  a  cost  of  $215 
million.  New  requirements  associated  with  automatic  landing  capabilities  increased 
the  RDT&E  costs  by  approximately  $120  million.  See  Figure  S.2  for  an  understand¬ 
ing  of  how  little  the  program  changed  after  Milestone  B  up  until  the  Nunn- McCurdy 
breach. 

For  procurement,  the  cost  adjustments  in  the  SARs  reveal  a  similar  pattern, 
with  a  few  costs  attributable  to  the  original  scope  of  JPALS  Inc.  1A  but  the  major¬ 
ity  of  changes  associated  with  restructuring  the  program.  Contract  cost  growth  and 
stretched  schedule  added  about  $19  million.  Procurement  decreased  by  $70  million 
because  of  the  reduction  in  shore  infrastructure  for  the  reduced  quantity  of  10  (27 
units  to  17  units).  Increasing  the  scope  of  the  current  JPALS  program  of  record  added 
$165  million  because  of  shifting  government  staff  from  later  increments  and  other 
small  restructuring  costs. 

There  were  occasional  downward  cost  adjustments,  which  are  not  directly  con¬ 
nected  to  the  root  causes  but  are  captured  in  the  analysis  process.  There  was  almost  $50 
million  in  downward  adjustments  in  the  SARs,  which  includes  estimating  (adjustment 
for  current  and  prior  escalation,  budget  cuts  from  DoD  and  Congress).  An  additional 
$71.6  million  was  associated  with  underestimating  systems  engineering,  integration, 
and  testing  (SEIT)  in  the  Inc.  1A  program  and  therefore  growth  in  the  cost  of  the 
contract.  This  $71.6  million  is  directly  related  to  meeting  the  original  requirements 
outlined  in  JPALS  Inc.  1A. 


7  CAPT  Darrell  D.  Lack,  U.S.  Navy,  “Program  Deviation  Report  Addendum  for  the  Joint  Precision  Approach 
and  Landing  System  Increment  1A  Program,”  memorandum  for  the  Linder  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics),  Department  of  the  Navy,  Program  Executive  Office,  Tactical  Aircraft  Program,  July 
24,  2012a. 
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Figure  S.2 

JPALS  Program  Cost  and  Planned  Quantity  Since  Milestone  B,  2008  Through  2013 
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SOURCES:  Department  of  Defense,  JPALS  Inc.  1A  SARs,  2008  through  2013. 
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After  categorizing  the  cost  variances  from  the  SARs,  we  summarized  the  cost 
driver  effect  at  the  unit  cost  level  and  associated  a  percentage  with  each.  This  gave  us  a 
better  understanding  of  the  increase  in  costs  based  on  spreading  those  costs  over  fewer 
units.  Figure  S.3  reflects  the  analysis  of  the  PAUC,  which  reinforces  the  notion  that 
stretched  schedule  and  contract  cost  growth  are  minor  drivers  of  JPALS  cost  growth 
(approximately  10  percent),  whereas  the  reduction  in  quantity  and  increased  scope 
accounted  for  over  85  percent  of  cost  growth. 

From  the  analysis  of  the  APUC,  it  appears  that  the  shore-based  facilities  that  were 
eliminated  during  the  restructuring  may  have  cost  more  per  unit  than  the  ship-based 
counterpart.  Therefore,  the  cost  associated  with  reducing  the  quantity  in  this  case  is 
unusually  negative.  Of  course,  the  smaller  quantity  did  mean  that  other  cost  vari¬ 
ances  had  a  greater  influence  on  unit  cost.  For  the  APUC,  contract  cost  growth  and 
stretched  schedule  have  a  minimal  effect  on  unit  cost  growth.  Increased  scope  shifted 
new  dollars  from  other  increments  and  therefore  accounts  for  90  percent  of  the  unit 
cost  growth  (see  Figure  S.4). 
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Figure  S.3 

Major  Driver  Is  Change  in  Scope,  But  Lower  Quantity  Spreads  RDT&E  Cost  over  Fewer  Units 
in  the  PAUC  Metric 
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SOURCES:  RAND  analysis;  Department  of  Defense,  JPALS  Inc.  1A  SARs,  December  2009  through  2013. 
NOTE:  Numbers  may  not  add  to  100  percent  because  of  rounding. 
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Figure  S.4 

About  90  percent  of  APUC  Growth  Is  Driven  by  Increased  Scope 
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SOURCES:  RAND  analysis;  Department  of  Defense,  JPALS  Inc.  1A  SARs,  December  2009  through  2013. 
NOTE:  Numbers  may  not  add  to  100  percent  because  of  rounding. 
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Root  Cause  Summary  of  Nunn-McCurdy  Breaches 

In  this  root  cause  analysis  on  JPALS  Inc.  1A  and  the  previous  root  cause  analyses 
on  other  weapon  systems,  PARCA  asked  RAND  to  investigate  the  main  causes  of 
Nunn-McCurdy  breaches.  RAND  discovered  that  breaches  most  frequently  occur  in 
the  following  three  areas:  planning,  changes  in  the  economy,  and  program  manage¬ 
ment.  Tables  S.3,  S.4,  and  S.5  provide  previously  identified  causes  of  Nunn-McCurdy 
breaches  along  with  JPALS  Inc.  1A  causes.  In  those  tables,  X  indicates  a  cause  of  a 
breach. 


Assessing  the  DoD  Acquisition  Portfolio 

For  purposes  of  oversight,  planning,  and  decisionmaking  regarding  development  and 
procurement,  policymakers  and  leaders  of  the  defense  acquisition  community  need  to 
be  able  to  describe  and  assess  the  status  and  performance  of  portfolios  of  major  defense 
acquisition  programs  (MDAPs)  over  time.  We  use  the  term  portfolio  to  refer  to  a  set 
of  acquisition  programs  that  can  be  grouped  together  because  they  share  a  certain 
characteristic  or  several  characteristics.  For  example,  the  helicopter  portfolio  is  a  set  of 
all  helicopter  programs.  Portfolios  can  be  designed  and  constructed  in  any  number  of 
ways  depending  on  the  interests  of  analysts.  For  instance,  portfolios  might  be  based 

Table  S.3 

Root  Causes  Stemming  from  Economic  Changes 


DDG- 

Navy 

JPALS 

Root  Cause 

WGS 

Apache 

1000 

JSF 

Excalibur 

ERP 

JTRS  Inc.  1A 

Increase  in  component  costs 

X 

X 

X 

X 

X 

Increase  in  labor  costs 

X 

X 

X 

Discontinued/decreased  production 

of  components 

Decreased  demand  for  similar 
technology  in  private  sector 
(economies  of  scale) 

X 

Decreased  demand  for  a  technology 

in  public  sector 

Inflation 

X 

X 

X 

X 

Production  delays 

X 

X 

X 

X 

Increase  in  procurement  quantities 

X 

X 

X 

Decrease  in  procurement  quantities 

X 

X 

X 

X 
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Table  S.4 

Root  Causes  Stemming  from  Planning 


Root  Cause 

WGS 

Apache 

DDG- 

1000 

JSF 

Navy 

Excalibur  ERP 

JTRS 

JPALS 
Inc.  1A 

Underestimate  of  baseline  cost 

X 

X 

X 

X 

X 

Underestimate  of  SEIT 

X 

Ambitious  scheduling  estimates 

X 

X 

X 

X 

Poorly  constructed  contractual 
incentives 

X 

X 

X 

Integration  of  new  technology  with 
legacy  systems 

X 

Immature  technologies 

X 

X 

X 

X 

Ill-conceived  manufacturing  process 

X 

Unrealistic  performance 
expectations 

X 

X  X 

Delay  in  awarding  contract 

X 

X 

X 

Insufficient  RDT&E 

X 

X 

X 

X 

X 

X 

Table  S.5 

Root  Causes  Stemming  from  Program  Management 


Root  Cause 

WGS 

Apache 

DDG- 

1000 

JSF 

Navy 

Excalibur  ERP 

JPALS 
JTRS  Inc.  1A 

Unanticipated  design,  manufacturing, 
and  technology  integration  issues 

X 

X 

X 

X 

X 

Lack  of  government  oversight  or  poor 
performance  by  contractor  personnel 

X 

X 

Inadequate  or  unstable  program 
funding 

X 

X 

X 

X 

X  X 

Unavailability  of  test  assets 

X 

Joint  status  of  program 
(e.g.,  management  complexity, 
budgeting  complexity,  and  multi¬ 
service/external  to  DoD  commitments) 

X 

Accounting  artifact 

X 
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on  the  types  of  programs  listed  in  the  Defense  Acquisition  Management  Information 
Retrieval  (DAMIR)  information  system  (all  force  application  or  logistics  programs, 
for  example).  Or  they  might  be  based  on  the  agency  or  service  that  owns  the  program. 
Such  portfolios  would  include  all  Army,  all  Navy,  or  all  DoD  programs.  Another  type 
of  portfolio  might  be  constructed  around  individual  contractors,  with  each  portfolio 
including  all  the  programs  currently  managed  by  key  DoD  contractors.  Finally,  a  port¬ 
folio  might  be  organized  to  include  all  programs  at  certain  points  in  the  program  life 
cycle,  past  Milestone  C  or  approaching  Milestone  B,  for  example. 

We  also  refer  to  subportfolios,  which  include  smaller  sets  of  MDAPs  with  simi¬ 
lar  characteristics.  The  analysis  of  subportfolios  enables  us  to  study  the  status  of  these 
smaller  sets  of  programs  in  more  detail  and  to  separate  their  status  from  the  status  of 
the  overall  portfolio  of  MDAPs  and  the  status  of  other  subportfolios. 

In  assessing  a  portfolio,  analysts  may  need  to  evaluate  the  maturity,  unit  cost 
growth,  or  schedule  performance  of  a  set  of  programs  and  then  summarize  the  status  of 
these  programs  using  clear  and  easy-to-understand  metrics  and  visualizations.  Analysts 
may  also  be  interested  in  how  a  portfolio’s  performance  changes  over  time,  how  its  per¬ 
formance  compares  to  other  portfolios,  and  even  how  the  composition  of  the  portfolio 
itself  may  be  evolving  (e.g.,  which  programs  have  been  completed  or  cancelled  and 
which  are  still  ongoing). 

The  analytic  approach  described  in  this  report  presents  a  methodology  and  set  of 
metrics  that  can  be  used  to  characterize  the  status  and  risk  of  portfolios  of  MDAPs  over 
time  and  across  commodity  types.  This  type  of  summary,  portfolio-level  assessment, 
differs  from  a  program-level  analysis,  which  would  focus  solely  on  a  single  program 
and  its  specific  performance  and  challenges.  A  summary  analysis  can  provide  a  broader 
insight  into  the  performance  of  a  set  of  acquisition  programs,  its  current  status,  future 
risks,  and  overall  performance  trends.  It  can  also  identify  programs  and  portfolios  that 
are  at  risk,  that  need  additional  investment,  or  that  are  performing  particularly  well. 
A  portfolio-level  assessment  can  also  help  acquisition  analysts  understand  the  ways  in 
which  a  given  portfolio  may  be  at  risk. 

Developing  a  Methodology 

The  methodology  that  we  developed  for  portfolio  assessment  included  the  following 
steps:  identify  objectives,  choose  a  portfolio  type,  select  data  and  metrics,  address  data 
anomalies  and  challenges,  calculate  metrics,  and  visualize  metrics.  The  first  step,  iden¬ 
tifying  the  objectives  of  the  portfolio  analysis,  lays  the  foundation  for  the  assessment 
itself  and  involves  defining  the  specific  set  of  questions  that  the  assessment  should 
answer.  These  questions  are  likely  to  reflect  both  the  analysts’  interests  and  policymak¬ 
ers’  priorities. 

Once  the  objectives  of  the  analysis  have  been  outlined,  the  next  step  is  to  select  a 
portfolio  or  portfolios  for  analysis.  Once  again,  this  choice  is  likely  to  respond  to  poli¬ 
cymakers’  interests  and  priorities.  Attention  may  focus  on  a  single  problematic  portfo- 
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lio,  or  it  may  involve  a  comparison  of  several  portfolios  to  guide  such  decisions  as  the 
allocation  of  funds  or  the  proper  amount  of  oversight  and  regulation. 

Perhaps  the  most  important  step  within  the  portfolio  analysis  process  is  the  selec¬ 
tion  of  metrics  and  data  sources  to  evaluate  the  status  of  the  portfolio  of  programs.  The 
identified  objectives  for  the  portfolio  analysis  should  frame  the  selection  of  data  and 
metrics  to  perform  the  assessment. 

After  reviewing  a  number  of  databases  and  data  sources,  the  team  concluded  that 
DAMIR  provided  the  most  appropriate  database  for  the  portfolio  analysis.  The  team 
also  assessed  various  data  sources  within  DAMIR,  considering  ease  of  access  for  the 
user  set,  consistency  and  validation  of  data,  and  the  regularity  of  data  collection  and 
presentation.  Taking  these  into  account,  the  team  concluded  that  SARs  would  provide 
the  primary  source  documents  for  data  inputs.  Congressionally  mandated,  the  SARs 
are  heavily  vetted  and  produced  annually,  with  occasional  quarterly  reports.8  When 
SARs  are  not  available  for  active  MDAP  programs,  the  team  used  the  Defense  Acqui¬ 
sition  Executive  Summary  (DAES)  reports  most  equivalent  to  the  December  SARs 
of  the  corresponding  year  to  fill  in  gaps  within  the  data.  Submitted  quarterly  to  the 
service-level  acquisition  databases  and  pulled  into  DAMIR,  DAES  include  not  only 
program  status  and  assessment  information  by  the  program  managers  but  also  inde¬ 
pendent  assessments  by  the  Office  of  the  Secretary  of  Defense  (OSD)  and  Joint  staff 
stakeholders.9  The  team  used  DAMIR,  SARs,  and  DAES  for  all  of  the  portfolio  cases, 
including  the  test  case  and  the  helicopter  and  satellite  portfolios. 

With  data  sources  selected,  the  team  moved  on  to  the  selection  of  metrics  to 
include  in  the  portfolio  analysis.  Our  portfolio  analysis  objectives — identify  and  char¬ 
acterize  cost  and  schedule  risks — guided  our  selection  of  program  and  portfolio  met¬ 
rics.  After  considering  a  wide  range  of  metrics  that  capture  different  aspects  of  port¬ 
folio  performance,  status,  and  risk,  we  agreed  on  a  set  of  core  metrics  that  met  our 
primary  objectives  and  that  we  believed  were  flexible  and  comprehensive  enough  to 
serve  as  the  foundation  for  an  effective  portfolio  analysis  framework.  These  metrics  and 
related  outcomes  and  indicators  are  listed  in  Table  S.6. 

Our  metrics  can  be  classified  into  several  different  groups.  First,  we  defined  both 
program-  and  portfolio-level  metrics.  Program-level  metrics  are  those  that  assess  the 
status  of  a  single  program,  whereas  portfolio-level  metrics  summarize  the  status  of 
an  entire  portfolio  of  programs.  Second,  we  include  both  descriptive  metrics  that 


8  10  U.S.C.  §2432  outlines  the  requirements  for  SAR  submissions,  which  are  mandatory  for  all  Acquisition 
Category  (ACAT)  I  programs.  Program  managers,  through  the  DAMIR  SAR  module  application,  prepare  SARs. 
Quarterly  reports  are  submitted  as  major  changes  are  observed  in  programs.  SARs  are  annual  unless  reporting 
an  APB  or  Nunn-McCurdy  breach,  in  which  case  a  program  submits  a  quarterly  SAR  (April  June,  and  Septem¬ 
ber).  Both  SAR  and  DAES  reporting  begin  at  Milestone  B,  unless  otherwise  initiated  by  the  Milestone  Decision 
Authority  (MDA). 

9  DAES  reporting  usually  begins  once  a  program  is  initiated  at  Milestone  B  and  ends  after  the  program  submits 
its  final  SAR.  DAES  is  reported  quarterly  in  three  groups  (A,  B,  and  C). 
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Table  S.6 

Program-  and  Portfolio-Level  Metrics 


Program-Level  Metric 

Outcomes/Indicators 

Measured 

Portfolio  Metric 

Descriptive-Type  Metrics 

Size  of  program  (dollars) 

Cost  outcomes,  political 
(external)  and  policy  risk 

Average,  median,  standard  deviation  in 
dollar  value 

Percentage  of  funds 
remaining 

Cost  and  schedule  outcomes, 
overall  risk 

Average,  median,  standard  deviation 
in  percentage  of  funds  remaining 

Percentage  of  time 
remaining 

Cost  and  schedule  outcomes, 
overall  risk 

Average,  median,  standard  deviation 
in  percentage  of  time  remaining 

Latest  milestone  Achieved 

Cost  and  schedule  outcomes, 
overall  risk 

Percentage  of  programs  passed  Milestone  B, 
Milestone  C 

Percentage  change  in 
quantity 

Cost  outcomes, overall  risk 

Average,  median,  standard  deviation 

"Churn":  average,  median,  standard 
deviation  of  absolute  value  of  percentage 
change  in  quantity 

Percentage  of  programs  with  a  quantity 
change  over  the  previous  year 

Performance-Type  Metrics 

Nunn-McCurdy  or  APB 
breaches 

Cost,  schedule,  performance 
outcomes,  political  risk 

Number  of  new  breaches 

Cumulative  total  breaches 

Percentage  of  programs  with  at  least  one 
breach 

Unit  cost 

growth:  percentage  change 
in  APUC  or  PAUC  (current 
and/or  original  baseline) 

Cost  outcomes,  funding  risk 

Average,  median,  standard  deviation  in 
PAUC/APUC  growth  from  current  baseline 

Percentage  of  programs  with  increase 
in  percentage  unit  cost  growth 

Distribution  of  percentage  unit  cost  growth 

RDT&E  and  procurement 
cost  growth 

Cost  outcomes,  technical  and 
requirements  risk 

Average,  median,  standard  deviation  in  cost 
growth 

Percentage  of  KPP  at  or 
above  threshold3 

Performance  outcomes, 
technical  and  requirements 
risk 

Average,  median,  standard  deviation 
of  percentage  KPP  at  or  above  threshold 

a  Unable  to  collect  consistent  data. 


simply  document  the  characteristics  of  the  portfolio  and  performance  metrics  that 
provide  deeper  insight  into  the  actual  status  of  the  portfolio  on  chosen  dimensions.  In 
other  words,  descriptive  metrics  provide  context  for  the  interpretation  of  performance 
metrics. 

Once  we  defined  our  metrics,  we  next  collected  the  data  needed  to  calculate 
them,  using  first  our  test  case  portfolio  and  later  the  helicopter  and  satellite  portfo¬ 
lios.  While  collecting  data  on  the  chosen  metrics,  we  came  across  a  number  of  data 
anomalies,  including  missing  data,  programs  that  split  or  changed  with  the  addition 
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of  new  blocks  or  modifications,  and  programs  that  rebaselined  during  the  assessment 
period  (affecting  unit  cost  and  quantity  calculations).  We  developed  a  set  of  assump¬ 
tions  to  deal  with  each  of  these  data  anomalies  and  then  carried  these  assumptions  for¬ 
ward  through  the  test  case  portfolio  and  then  the  helicopter  and  satellite  portfolios.  To 
maintain  an  auditable  product,  analysts  performing  their  own  portfolio  analysis  will 
need  to  carefully  track  these  anomalies  and  the  assumptions  or  measures  they  took  to 
address  these. 

After  collecting  data  on  each  program  within  our  portfolios  of  interest,  we  then 
calculated  the  associated  portfolio  metrics.  Because  our  intention  was  to  develop  a 
repeatable  methodology  that  could  be  applied  to  portfolios  of  all  kinds,  we  spent  con¬ 
siderable  time  developing  a  generalizable  computer  program  that  can  easily  calculate 
portfolio  metrics.10  This  same  program  can  be  used  to  calculate  metrics  for  the  satellite 
and  helicopter  portfolios  and  will  be  equally  applicable  to  other,  larger  portfolios  with 
more  years  of  data. 

As  noted  in  Table  S.6,  the  portfolio  metrics  that  we  calculated  included  mean, 
median,  and  standard  deviation  of  such  quantities  as  percentage  unit  cost  growth, 
quantity  change,  total  program  value,  and  percentage  of  RDT&E  and  procurement 
funds  remaining.  We  also  counted  the  number  and  types  of  breaches  and  the  number 
of  programs  past  key  points  in  development,  such  as  Milestones  B  and  C.  Each  portfo¬ 
lio  metric  gave  us  a  unique  window  on  the  status  of  the  portfolio.  Furthermore,  these 
metrics  can  be  combined  to  provide  a  more  holistic  view  of  portfolio  status. 

In  addition  to  looking  at  static  portfolio  metrics  to  assess  the  status  of  our  port¬ 
folios  in  each  year,  we  also  hoped  to  compare  across  years  to  understand  how  the 
portfolio  had  changed  over  time.  This  involved  not  only  calculating  metrics  but  also 
determining  whether  changes  we  observed  between  years  were  statistically  significant 
(meaning  that  we  can  say  with  some  confidence,  usually  95  percent,  that  the  observed 
change  is  different  from  zero)  and  substantively  meaningful  (or  sizable  enough  to  sug¬ 
gest  a  meaningful  change  over  time).  To  address  statistical  significance,  we  used  t-tests 
and  fixed-effects  models.* 11  For  substantive  importance,  we  used  a  measure  known  as 


10  The  generalized  computer  program  was  developed  in  STATA,  a  statistical  package  used  to  conduct  data 
analysis. 

11  For  the  test  case  with  only  two  years  of  data,  we  relied  on  paired  t-tests  for  unweighted  data.  The  paired  t-test 
takes  this  program  consistency  into  account.  The  t-test  compares  the  difference  between  the  means  of  the  two 
samples  (2010  and  2012)  to  the  variance  within  each  sample  independently,  taking  into  account  the  number  of 
observations.  When  this  ratio  value  is  large  (greater  than  two),  then  the  difference  between  the  two  years  is  said  to 
be  statistically  significant  and  is  unlikely  to  be  due  to  chance  or  to  noise  in  the  data.  For  the  satellite  and  helicop¬ 
ter  portfolios,  however,  we  needed  a  different  way  to  assess  statistical  significance  because  we  had  so  many  more 
years  of  data.  In  these  instances,  we  conducted  a  number  of  tests.  First,  we  used  a  fixed-effects  model,  which  is 
able  to  account  for  program-specific  characteristics  (essentially  includes  a  control  variable  for  each  program  in  the 
analysis).  The  fixed-effects  model  allows  us  to  determine  whether  observed  differences  in  portfolio  performance 
between  any  two  years  are  statistically  significant,  meaning  that  we  can  say  with  some  certainty  that  they  are  dif¬ 
ferent  from  zero  or  that  the  change  is  a  real  change  in  the  trend  and  is  not  simply  noise  in  the  data. 
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Table  S.7 

Visualizations  and  Information  Provided 


Visualization 

Type  of  Information  Provided 

Histograms  and  bar  charts 

Distribution  of  programs 

Trends  in  numerical  metrics 

Box-and-whisker  plots 

Medians  and  change  over  time 

Range  and  spread  of  data  (with  or  without  extreme  values) 

"Heat"  map 

Year-on-year  changes  across  the  portfolio  for  one  metric 

More  detailed  view  of  program-level  information 

Bubble  diagrams 

Two  metrics  displayed  simultaneously 

A  measure  of  a  program's  "influence"  in  portfolio 

One  way  to  visualize  a  weighted  metric 

Radar  charts 

Aggregate  view  of  several  different  metrics  at  one  or  several  points  in  time 

Cohen’s  D,  which  also  compares  the  difference  between  the  means  of  the  two  sample 
years  and  the  differences  or  variation  within  each  year. 

The  final  step  in  our  portfolio  analysis  was  the  construction  of  visualizations  that 
can  be  used  to  summarize  each  portfolio  metric.  Table  S.7  lists  the  visualizations  used 
and  the  information  that  each  is  intended  to  provide.  In  the  report,  we  provide  addi¬ 
tional  detail  about  the  value  and  uses  of  each  type  of  visualization.  We  included  several 
types  of  visualizations  because  each  provides  a  specific  type  of  information  and  serves 
a  specific  purpose  in  the  analysis.  Our  visualizations  include  those  that  capture  a  single 
point  in  time,  those  that  compare  across  years,  and  those  that  capture  several  metrics 
in  a  single  picture. 

Application  of  the  Methodology:  Examining  Metrics 

After  describing  in  detail  our  approach  to  portfolio  analysis,  we  next  apply  the  method¬ 
ology  to  the  helicopter  and  satellite  portfolios  from  2002  to  2012.  Our  focus  remains 
on  demonstrating  and  validating  the  methodology  and  the  selection  of  metrics  rather 
than  on  conducting  a  comprehensive  assessment  of  the  portfolio.  By  discussing  the 
application  of  our  methodology  in  more  depth,  we  are  able  to  highlight  the  value  of 
metrics  included  in  the  assessment,  how  they  can  be  analyzed  and  interpreted,  and 
how  visualizations  can  be  used  to  summarize  the  results  of  the  analysis.  Our  initial 
application  of  the  methodology  provides  the  “what”  of  the  portfolio  analysis:  What  are 
the  composition,  the  characteristics,  and  the  performance  indicators  of  the  portfolio? 
In  our  analysis,  we  focused  primarily  on  cost  and  schedule  performance  indicators  as 
central  to  our  assessments  of  the  helicopter  and  satellite  portfolios.  However,  we  also 
considered  the  composition  of  the  portfolio  and  how  it  evolved  over  the  11-year  period 
in  our  analysis,  the  maturity  of  the  portfolio,  and  trends  in  Nunn-McCurdy  and  APB 
breaches.  For  each  metric  considered  in  the  analysis,  we  also  present  and  interpret 
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visualizations  of  the  data.  We  include  a  number  of  multimetric  charts  that  summa¬ 
rize  performance  across  several  dimensions  simultaneously.  Metrics  that  we  considered 
along  with  a  summary  of  their  contribution  to  the  portfolio  analysis  include  milestones 
achieved,  composition,  program  value,  quantity  changes,  and  number  and  type  of 
breaches. 

In  addition  to  considering  the  performance  of  each  portfolio  of  programs  over 
time,  we  also  compare  the  performance  of  the  two  portfolios  as  yet  another  way  of 
highlighting  how  the  metrics  included  in  the  analysis  can  be  used  and  interpreted. 
The  two  portfolios  show  some  differences  as  well  as  some  similarities.  The  helicopter 
portfolio  appears  to  show  a  gradually  maturing  (and  possibly  aging)  portfolio  that 
experiences  some  improvement  in  unit  cost  growth  and  number  of  breaches  over  the 
assessment  period.  The  satellite  portfolio,  on  the  other  hand,  has  more  year-to-year 
changes  in  composition  and  somewhat  less  change  in  overall  portfolio  status  over  the 
assessment  window.  However,  the  satellite  portfolio  still  has  some  programs  that  expe¬ 
rience  extended  periods  of  poor  performance,  especially  between  2007  and  2010.  The 
most  significant  similarity  between  the  two  portfolios  is  that  both  seem  be  improving 
on  certain  metrics  and  at  the  aggregate  level  in  recent  years. 

One  important  lesson  from  our  application  of  our  methodology  to  our  two  sample 
portfolios  is  the  importance  of  viewing  multiple  metrics  from  different  perspectives  to 
better  understand  patterns  and  trends  in  portfolio  status  and  future  risk.  Single  metrics 
do  not  provide  a  complete  picture  of  portfolio  performance.  The  interpretation  of  per¬ 
formance  metrics  must  also  be  done  in  the  context  of  the  descriptive  metrics,  especially 
awareness  of  programs  moving  in  and  out  of  the  portfolio,  the  potential  for  one  or  two 
programs  to  dominate  the  results  of  portfolio  metrics,  and  the  size  of  the  program  and 
funds  remaining  as  a  measure  of  future  portfolio  risk.  A  second  important  point  that 
should  be  factored  into  any  assessment  of  program  or  portfolio  performance  is  that  the 
utility  of  the  analysis  depends  fundamentally  on  the  quality  of  the  data.  An  assessment 
using  data  that  has  missing  data  points,  errors,  or  other  problems  can  produce  results 
that  are  misleading  and  can  lead  to  misinformed  decisions. 

Application  of  the  Methodology:  Assessing  the  Helicopter  Portfolio 

The  analysis  of  our  selected  metrics  and  visualizations  allows  for  a  repeatable  and  audit- 
able  process  to  track  portfolio  status  indicators  over  time  or  across  various  portfolios. 
Inevitably,  however,  a  policymaker  or  decisionmaker  will  want  to  delve  into  “why” 
portfolio  trends  or  patterns  have  emerged  to  adjust  his  or  her  policies  or  priorities 
accordingly  and  to  understand  whether  near-term  action  can  mitigate  future  port¬ 
folio  risk.  The  portfolio  analysis  methodology  outlined  in  this  report  can  be  used  to 
answer  these  additional  questions  and  guide  policymaker  decisions.  After  examining 
our  metrics  and  visualizations  in  the  context  of  the  helicopter  and  satellite  portfolios, 
we  took  our  analysis  of  the  helicopter  portfolio  one  step  further,  offering  an  abbrevi¬ 
ated  example  of  how  a  complete  portfolio  analysis  might  proceed.  Although  we  did  not 
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conduct  a  comprehensive  root  cause  analysis  for  each  of  the  programs  in  the  portfolio, 
we  did  use  general  root  cause  methods  to  pinpoint  cost  growth  events  for  the  portfolio 
and  to  identify  both  the  individual  program  performance  issues  and  external  forces 
or  trends  that  may  have  driven  those  events.  The  major  categories  of  potential  policy¬ 
makers’  questions  include  greater  contextual  understanding,  further  examination  into 
the  portfolio  analysis  cost  and  schedule  risk  objectives,  and  further  potential  analysis 
inspired  by  the  observed  trends  within  the  portfolio  analysis. 

In  our  more  detailed  assessment,  we  delve  more  deeply  into  issues  such  as  the  rea¬ 
sons  for  changes  in  the  portfolio’s  composition  and  maturity,  the  drivers  and  implica¬ 
tions  of  rates  of  program  spending  and  changes  in  percentage  of  funds  remaining,  pos¬ 
sible  explanations  for  observed  changes  in  unit  cost  growth  and  the  potential  for  future 
cost  growth,  the  effects  of  rebaselines,  and  trends  in  Acquisition  Program  Baseline  and 
Nunn-McCurdy  breaches.  Finally,  we  use  multimetric  charts  to  offer  a  more  holistic 
assessment  of  the  portfolio.  Although  our  assessment  of  the  helicopter  portfolio  is  by 
no  means  complete,  a  deeper  assessment  of  the  portfolio  suggests  the  ways  in  which  our 
methodology  can  be  used  to  comprehensively  study  a  portfolio  of  programs  and  how 
our  initial  set  of  metrics  can  spawn  a  range  of  other  lines  of  inquiry  that  can  be  used  to 
answer  policymaker  questions  and  inform  their  decisions. 
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AAS 

Armed  Aerial  Surveillance 

ACAT 

Acquisition  Category 

ACLS 

Automatic  Carrier  Landing  System 

ACS 

Auxiliary  Crane  Ship 

ADM 

Acquisition  Decision  Memorandum 

AEHF 

Advanced  Extremely  High  Frequency 

AFROCC 

Air  Force  Requirements  for  Operational  Capability  Council 

AHE 

E-2D  Advanced  Hawkeye 

AIG 

Aircraft  Integration  Guide 

AIM 

Acquisition  Information  Management 

AIR 

Acquisition  Information  Repository 

AMF 

Airborne  and  Maritime/Fixed  Station 

AMRAAM 

Advanced  Medium-Range  Air-to-Air  Missile 

APB 

Acquisition  Program  Baseline 

AoA 

Analysis  of  Alternatives 

APB 

Acquisition  Program  Baseline 

APUC 

average  procurement  unit  cost 

ARH 

Armed  Reconnaissance  Helicopter 

AS 

Acquisition  Strategy 

ASD  (AT&L)  Assistant  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 
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ASN/RDA 

Assistant  Secretary  of  the  Navy  (Research,  Development  and 
Acquisition) 

ASR 

acquisition  stability  reserve 

ATNAVICS 

Air  Traffic  Navigation,  Integration,  and  Coordination  System 

AVTK 

Avionics  Test  Kit 

BY 

base  year 

C3I 

command,  control,  communications,  and  intelligence 

C4I 

command,  control,  communications,  computers,  and  intelligence 

CAAS 

Common  Avionics  Architecture  System 

CAPE 

Cost  Assessment  and  Program  Evaluation 

CDD 

Capability  Development  Document 

CCDR 

Contractor  Cost  Data  Report 

CDR 

Critical  Design  Review 

Chem-Demil- 

ACWA 

Chemical  Demilitarization-Assembled  Chemical  Weapons 

Alternatives 

CNO 

Chief  of  Naval  Operations 

COTS 

commercial  off-the-shelf 

CSB 

Configuration  Steering  Board 

CV 

Carrier  Variant 

CVN 

Navy  Nuclear  Aircraft  Carrier 

cvw 

Carrier  Air  Wing 

DAB 

Defense  Acquisition  Board 

DACIMS 

Defense  Automated  Cost  Information  Management  System 

DAE 

Defense  Acquisition  Executive 

DAES 

Defense  Acquisition  Executive  Summary 

DAMIR 

Defense  Acquisition  Management  Information  Retrieval 

DCARC 

Defense  Cost  and  Resource  Center 

DDG 

Guided  Missile  Destroyer 

Abbreviations 
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DH 

decision  height 

DoD 

Department  of  Defense 

DoDI 

Department  of  Defense  Instruction 

DoT 

Department  of  Transportation 

DUSD 

Deputy  Under  Secretary  of  Defense 

EDM 

engineering  development  model 

EMD 

engineering  and  manufacturing  development 

ESG 

Expeditionary  Strike  Group 

EVA 

Earned  Value  Analysis 

EVM 

Earned  Value  Management 

FAA 

Federal  Aviation  Administration 

FAB-T 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals 

FADEC 

Full  Authority  Digital  Engine  Control 

FBW 

Fly-By-Wire 

FCR 

Fire  Control  Radar 

FMS 

Foreign  Military  Sales 

FOC 

Full  Operational  Capability 

FPIF 

Fixed  Price  Incentive  Firm 

FRP 

Federal  Radionavigation  Plan;  also  full-rate  productioi 

FY 

fiscal  year 

GAO 

Government  Accountability  Office 

GBAS 

Ground-Based  GPS  Augmentation  System 

GBS 

Global  Broadcasting  Service 

GEO 

geosynchronous  earth  orbit 

GNSS 

Global  Navigation  Satellite  System 

GOTS 

government  off-the-shelf 

GPS 

Global  Positioning  System 
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GPS  OCX 

Global  Positioning  System  Next  Generation  Operational  Control 
System 

HMS 

Handheld,  Manpack  and  Small  Form  Fit 

HUD 

Head  up  Display 

IAMD 

Integrated  Air  and  Missile  Defense 

IBR 

Integrated  Baseline  Review 

ICAO 

international  civil  aviation  organization 

ICD 

Initial  Capabilities  Document 

ICE 

Independent  Cost  Estimate 

ICLS 

Instrument  Carrier  Eanding  System 

ILS 

Instrument  Eanding  System 

IOC 

initial  operational  capability 

IQR 

interquartile  range 

ISR 

Intelligence,  surveillance,  and  reconnaissance 

JASSM 

Joint  Air-to-Surface  Standoff  Missile 

JCATD 

JPALS  Common  Avionics  Technology  Development 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

JLENS 

Joint  Land  Attack  Cruise  Missile  Defense  Elevated  Netted  Sensor 
System 

JPALS 

Joint  Precision  Approach  and  Landing  System 

JROC 

Joint  Requirements  Oversight  Council 

JSF 

Joint  Strike  Fighter 

JTN 

Joint  Tactical  Network 

JTRS 

Joint  Tactical  Radio  System 

KPP 

key  performance  parameter 

KSA 

key  system  attribute 

LAAS 

Local  Area  Augmentation  System 

LCS 

Littoral  Combat  Ship 
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LHA/D 

Landing  Helicopter  Assault 

LHD 

Landing  Helicopter  Dock 

MLS 

Microwave  Landing  System 

MNS 

Mission  Needs  Statement 

LPD 

Landing  Platform  Dock 

LRIP 

low-rate  initial  production 

LUH 

light  utility  helicopter 

MAIS 

Major  Acquisition  Information  System 

MAR 

Monthly  Acquisition  Report 

MATCALS 

Marine  Air  Traffic  Control  and  Landing  System 

MDA 

Milestone  Decision  Authority 

MDAP 

major  defense  acquisition  program 

MILCON 

military  construction 

MMLS 

Mobile  Microwave  Landing  System 

MNS 

Mission  Needs  Statement 

MoA 

Memorandum  of  Agreement 

MS 

milestone 

MUOS 

Mobile  User  Objective  System 

NAS 

National  Airspace  System;  also,  Naval  Air  Station 

NASA 

National  Aeronautics  and  Space  Administration 

NATO 

North  Atlantic  Treaty  Organization 

NAVAID 

navigational  aid 

NDAA 

National  Defense  Authorization  Act 

NESP 

Navy  EHF  Satellite  Communications  Program 

NPOESS 

National  Polar-Orbiting  Operational  Environmental  Satellite  System 

O&M 

operations  and  maintenance 

O&S 

operations  and  sustainment 
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OA 

operational  assessment 

OASIS 

Organic  Airborne  Surface  Influence  Sweep 

ORD 

Operational  Requirements  Document 

OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

Operational  Test  &  Evaluation 

OTH 

over  the  horizon 

OTS 

off-the-shelf 

PAL 

precision  approach  and  landing 

PALC 

precision  approach  and  landing  capability 

PAR 

Precision  Approach  Radar 

PAR/ILS 

Precision  Approach  Radar  and  Hybrid  Instrument  Landing  System 

PARCA 

Performance  Assessments  and  Root  Cause  Analyses 

PAUC 

program  acquisition  unit  cost 

PDR 

Preliminary  Design  Review;  also,  Program  Deviation  Report 

P.L. 

Public  Law 

PLCCE 

Program  Life  Cycle  Cost  Estimates 

PNO 

program  number 

PNT 

positioning,  navigation,  and  timing 

POM 

Program  Objective  Memorandum 

R3B 

Resources  and  Requirements  Review  Board 

R&D 

research  and  development 

RDAIS 

Research,  Development,  and  Acquisition  Information  System 

RDT&E 

research,  development,  test,  and  evaluation 

RMS 

Remote  Minehunting  System 

RNAV 

area  navigation 

RVR 

runway  visual  range 

SAF/AQ 

Secretary  of  the  Air  Force  for  Acquisition 
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SAR 

SBAS 

SBIRS-High 

SBSS 

SCP 

SDD 

SEIT 

SFR 

SM 

SMART 

SRR 

SSN 

STAR 

STOVL 

SV 

TACAN 

TMS 

TRL 

TRN 

TRR 

TSAT 

UAS 

UAV 

UCLASS 

UK 

use 

USD  (AT&L) 


Selected  Acquisition  Report 
Space-Based  Augmentation  System 
Space-Based  Infrared  System-High 
space-based  surveillance 
Service  Cost  Position 
System  Development  and  Demonstration 
systems  engineering,  integration,  and  testing 
System  Functional  Review 
statute  mile 

System  Metrics  and  Reporting  Tool 
System  Requirements  Review 
nuclear  submarine 
System  Threat  Assessment  Report 
short  takeoff/vertical  landing 
satellite  vehicle 

Tactical  Air  Navigation  System 
Thermal  Management  System 
technology  readiness  level 
Terrain  Relative  Navigation 
Test  Readiness  Review 
Transformational  Satellite  System 
Unmanned  Aircraft  System 
unmanned  aerial  vehicle 
pilotless  aircraft 
United  Kingdom 
U.S.  Code 
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VTUAV 

WAAS 

WGS 

WSARA 


Vertical  Takeoff  and  Landing  Tactical  Unmanned  Air  Vehicle 

Wide  Area  Augmentation  System 

Wideband  Global  SATCOM 

Weapon  Systems  Acquisition  Reform  Act 
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AMS-2100 

AN/FPN-67 

AN/SPN-41 

AN/TPN-31 

ASR/PAR 

ATNAVICS 

AV-8B 

C-2A 


Series  2100  Instrument  Landing  System  (ILS);  provides  Category  I, 
II,  and  III  performance  that  comes  in  multiple  configurations  with  a 
variety  of  antenna  arrays  that  can  be  upgraded  in  the  field. 

Fixed  Base  Precision  Approach  Radar;  provides  aircraft  position 
information  to  aid  in  landing  operations  in  adverse  weather  and  low 
visibility  conditions. 

Transmitting  Set  Used  for  Landing;  provides  all-weather  instrument 
approach  guidance  from  the  ship  and  works  with  aircraft  equipped 
with  the  AN/ARA-63  receiver  group.  Often  used  as  the  ship’s 
Instrument  Carrier  Landing  System  (ICLS)  to  provide  azimuth  and 
elevation  alignment  information. 

Radar  Approach  Control  System;  used  for  controlling  and  landing 
fixed-  and  rotary-wing  aircraft  requirements.  The  only  fully 
autonomous,  ICAO/NAS -compliant  system  that  is  transportable  in 
a  single  C-130  aircraft. 

Airport  Surveillance  Radar/Precision  Approach  Radar;  a  type  of 
radar  instrument  approach  provided  with  active  assistance  from  air 
traffic  control.  The  only  additional  equipment  need  is  a  functioning 
radio  transmitter  and  receiver. 

Air  Traffic  Navigation,  Integration,  and  Coordination  System; 
consists  of  a  radar  vehicle,  an  operations  control  vehicle,  and 
two  XM1102  tactical  power  generator  (lOKw)  trailers.  Used  in 
conjunction  with  AN/TPN-31. 

AV-8B  Harrier  II;  a  single-engine  and  fixed-wing  aircraft  that  has 
vertical/short  takeoff  and  landing. 

Grumman  C-2A  Greyhound;  a  twin-engine,  high-wing  cargo 
aircraft,  designed  to  carry  supplies  and  logistical  support.  Serves  as 
the  Navy’s  carrier-onboard-delivery  aircraft. 
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CH-53K 

CVW 

DH 

E-2D 

EA-18G 

F/A-18 

F/A-18E/F 

Fk-35B/C 

FAA 


Sikorsky  CH-53K  Super  Stallion;  a  large,  heavy-lift  cargo  helicopter 
currently  being  developed  by  Sikorsky  Aircraft  for  the  Marine 
Corps.  It  will  be  fully  shipboard-compatible  and  capable  of 
operating  from  austere  and  remote  forward  operating  bases. 

Carrier  Air  Wing;  a  naval  aviation  organization  composed  of  several 
aircraft  squadrons  and  detachments  of  various  types  of  fixed-wing 
and  rotary-wing  aircraft. 

Decision  Height;  a  specified  height  in  the  precision  approach  or 
approach  with  vertical  guidance  at  which  a  missed  approach  must 
be  initiated  if  the  required  visual  reference  to  continue  the  approach 
has  not  been  established. 

Northrop  Grumman  E-2  Hawkeye;  an  all-weather,  carrier-capable 
tactical  airborne  early  warning  (AEW)  aircraft.  Provides  improved 
battle  space  target  detection  and  situational  awareness. 

Airborne  Electronic  Attack  (AEA)  Aircraft;  currently  being 
delivered  to  the  U.S.  Navy,  it  is  derived  from  the  combat  proven 
F/A-18F  aircraft  and  incorporates  advanced  AEA  avionics,  which 
suppresses  enemy  air  defenses  (SEAD)  and  nontraditional  electronic 
attack  operations. 

F/A-18  Hornet;  a  twin-engine,  all-weather,  day  or  night  Marine  jet 
that  can  be  used  for  fighter  escort,  enemy  air  defense  suppression, 
reconnaissance,  air  control,  and  close  air  support. 

F/A-18E/F  Super  Hornet;  a  twin-engine  supersonic,  all- 
weather  carrier-capable  multirole  combat  jet.  Provides  enhanced 
interoperability,  total  force  support  for  the  combatant  commander 
and  for  the  troops  on  the  ground. 

F-35B  Short  Takeoff/Vertical  Landing  (STOVL)/  F-35C  Carrier 
Variant  (CV);  aircraft  with  supersonic,  low-observable  stealth 
fighters  used  to  execute  multirole  missions  and  to  support  F-35 
sustainment  technologies. 

Federal  Aviation  Administration;  the  national  aviation  authority  of 
the  United  States.  An  agency  of  the  Department  of  Transportation, 
it  has  authority  to  regulate  and  oversee  all  aspects  of  American  civil 
aviation. 
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Fire  Scout 


FPN-62 

FPN-63 

FRP 

GBAS 

GPN-22 

GPS 

GRN-29 

H-60R/S 

ILS 

JPALS 


Northrop  Grumman  MQ-8  Fire  Scout;  an  intelligence,  surveillance 
and  reconnaissance  (ISR)  vertical  takeoff  and  landing  tactical 
unmanned  air  vehicle  (VTUAV).  The  baseline  MQ-8B  can 
accomplish  missions  including  over-the-horizon  (OTH)  tactical 
reconnaissance,  classification,  targeting  and  laser  designation,  and 
battle  management. 

AN/FPN-62  Radar  Set;  a  ground  controlled  approach  radar  set 
that  uses  split  elevation  and  azimuth  antennas.  The  antennas  are 
reflectors  with  electronically  scanned  phased-array  feeds. 
AN/FPN-63(V)  Common  Precision  Approach  Radar;  provides  talk- 
down  capabilities  to  land  military  and  civil  aircraft  during  reduced 
meteorological  conditions.  It  also  provides  in-garrison  training  for 
controllers  and  pilots  in  support  of  aircraft  carrier  talk-down  final 
approach  landings. 

Federal  Radionavigation  Plan;  the  official  source  of  positioning, 
navigation,  and  timing  (PNT)  policy  and  planning  for  the  federal 
government. 

Ground-Based  Augmentation  System;  provides  differential 
corrections  and  integrity  monitoring  of  Global  Navigation  Satellite 
System  (GNSS).  GBAS  provides  navigation  and  precision  approach 
service  in  the  vicinity  of  the  host  airport. 

AN/GPN-22  PAR;  radar  intended  for  unattended  use  in  fixed-base, 
high-density  air  traffic  control  system  operations  under  all  weather 
conditions. 

Global  Positioning  System;  a  radio  navigation  system  that  allows 
land,  sea,  and  airborne  users  to  determine  their  exact  location, 
velocity,  and  time  in  all  weather  conditions. 

AN/GRN-29(v)  Instrument  Landing  System;  a  solid  state 
instrument  landing  system  power  supply. 

The  Sikorsky  MH-60S  Knighthawk  and  the  MH-60R  Seahawk; 
part  of  the  Navy  submarine  hunter  and  anti-surface  warfare 
helicopter  fleet. 

Instrument  Landing  System;  a  system  of  transmitters  that  provide 
the  lateral  and  vertical  guidance  necessary  to  fly  a  precision 
approach. 

Joint  Precision  Approach  and  Landing  System;  a  GPS/Inertial 
Navigation  System  that  will  provide  a  rapidly  deployable,  adverse 
weather,  adverse  terrain,  day-night  precision  approach  and  landing 
capability  for  all  DoD  ground  and  airborne  systems. 
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LAAS 


LHA/D 


MK-20 

MK-20A 

MPN-14K 


MPN-25 


MV-22 


SBAS 


SEIT 


SPN-35/41 


Local  Area  Augmentation  System;  the  FAA  version  of  the  GBAS 
and  is  based  on  a  single  GPS  reference  station  facility  located  on  the 
property  of  the  airport  being  serviced. 

Amphibious  Assault  Ships;  Navy  ships  that  project  power  and 
maintain  presence  by  serving  as  the  cornerstone  of  the  Amphibious 
Readiness  Group  (ARG)/Expeditionary  Strike  Group  (ESG). 
Fixed-base  ILS. 

Fixed-base  ILS;  distance  measuring  system. 

AN/MPN-14K  Landing  Control  Central;  the  radar  unit  is  used  by 
air  traffic  controllers  to  identify,  sequence,  and  separate  participating 
aircraft;  provide  final  approach  guidance,  guidance  through  air 
defense  corridors  and  zones,  and  coordinate  identification  and  intent 
with  local  air  defense  units  at  assigned  airports,  air  bases,  and  bare 
bases. 

AN/MPN-25;  systems  provide  a  critical  capability  to  the  U.S.  Air 
Force  in  carrying  out  its  mission.  The  Mobile  Microwave  Landing 
System  (MMLS)  radiates  a  precision  signal  to  guide  aircraft  to  the 
runway,  much  like  civilian  instrument  landing  systems  found  at 
domestic  airports,  but  its  frequency  and  aircraft  are  unique. 

MV-22  Osprey;  Marine  Corps  tiltrotor  aircraft,  which  is  designated 
as  the  “baseline”  variant,  is  a  vertical/short  takeoff  and  landing, 
medium-lift  assault,  self-deployment,  and  sustained  land-operations- 
capable  air  vehicle. 

Space-Based  Augmentation  System;  the  FAA  has  developed  a  Wide 
Area  Augmentation  System  (WAAS)  to  provide  accurate  positioning 
to  the  aviation  industry — a  service  that  is  free  to  all  other  civilian 
users  and  markets  in  Central  and  North  America.  This  service  falls 
into  the  greater  category  of  Space-Based  Augmentation  System 
(SBAS). 

Systems  engineering,  integration,  and  testing;  provides 
programmatic  insight  to  optimize  and  integrate  schedules,  track 
technical  baselines,  issue  tracking  and  resolution,  test,  and 
operations  across  a  program. 

The  AN/SPN-35C/41A  Aircraft  Control  Approach  Central;  an 
upgrade  to  the  AN/SPN-35B  that  provides  fleet  air  traffic  controllers 
an  all-weather  precision  approach  radar  system  for  safe  landing  of 
aircraft  onboard  amphibious  assault  ships. 
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SPN-42 


SPN-46/41 


TACAN 

(URN-25) 

TMS 

TPN-22 

TPN-25 

TRN-4 


UCLASS 

UH-1Y/Z 
UK  CVF 
WAAS 


AN/SPN-42;  a  computerized  Automatic  Carrier  Landing  System 
(ACLS)  radar  that  provides  precise  control  of  aircraft  during  their 
final  approach  and  landing.  The  equipment  can  automatically 
acquire,  control,  and  land  a  suitably  equipped  aircraft  on  aircraft 
carriers  under  severe  ship  motion  or  weather  conditions. 
AN/SPN-46(V)/AN-SPN-4l;  the  ACLS  is  a  precision  approach 
landing  system  (PALS)  that  provides  electronic  guidance  to  carrier- 
based  aircraft  and  allows  them  to  land  in  all-weather  conditions 
with  no  limitations  due  to  low  ceiling  or  restricted  visibility.  The 
AN/SPN-46  is  the  fleet’s  only  fully  automated,  all-weather  approach 
landing  aid  for  carrier  aircraft. 

Tactical  Air  Navigation  System;  a  line-of-sight,  beacon-type,  air 
navigation  aid  that  provides  slant  range,  bearing,  and  identification 
information  to  TACAN-equipped  aircraft  in  determining  the 
aircraft  position. 

Thermal  Management  Systems;  used  for  aircraft  landing  on  ship 
decks. 

AN/TPN-22  Precision  Approach  Radar  (PAR);  a  transportable, 
computerized,  pencil  beam,  three-dimensional  radar.  The  system  is  a 
track-while-scan  radar  used  for  landing  tactical  aircraft. 

Air  Force  mobile  PARs  include  TPN-25;  a  solid-state  precision  radar 
that  is  capable  of  simultaneously  tracking  and  controlling  six  aircraft 
to  a  specific  range. 

Terrain  Relative  Navigation  (TRN);  used  to  augment  inertial 
navigation  by  providing  position  or  bearing  measurements  relative 
to  known  surface  landmarks.  A  vehicle’s  position  is  estimated  by 
comparing  terrain  measurements  (e.g.,  sonar  or  altimeter)  with  a 
terrain  map. 

Unmanned  Carrier- Launched  Airborne  Surveillance  and  Strike; 
a  program  that  will  develop  a  long  duration,  carrier-based  UAS 
originally  capable  of  operations. 

UH-1Y  Fluey/Venom;  a  twin-engine,  medium-size  utility  attack 
helicopter,  part  of  the  Marine  Corps  H-l  Upgrades  program. 

Queen  Elizabeth— class  (formerly  the  CV  Future  or  CVF  project);  a 
class  of  two  aircraft  carriers  being  built  for  the  UK  Royal  Navy. 
Wide  Area  Augmentation  System;  an  accurate  navigation  system 
developed  for  civil  aviation.  It  provides  service  for  all  classes  of 
aircraft  in  all  phases  of  flight,  including  en  route  navigation,  airport 
departures,  and  airport  arrivals. 


CHAPTER  ONE 


Introduction 


Background  and  Purpose 

This  combined  report  deals  with  two  issues.  One  is  an  analysis  of  the  Nunn-McCurdy 
breach  that  occurred  in  the  Joint  Precision  Approach  and  Landing  System  Incre¬ 
ment  1A  (JPALS  Inc.  1A)  acquisition  program.1  This  particularly  ambitious  program 
involved  not  only  the  Army,  the  Navy,  and  the  Air  Force  but  also  civilian  agencies,  and 
it  had  international  implications  because  it  involved  the  widely  used  Instrument  Land¬ 
ing  System  (ILS) — a  system  dating  back  to  the  late  1920s  that  relies  on  radio  beams 
to  guide  pilots  to  the  runway  when  visibility  is  reduced.  The  advent  of  the  Global  Posi¬ 
tioning  System  (GPS)  provided  an  alternative  to  the  ILS,  but  developing  a  GPS-based 
system  that  met  the  needs  of  diverse  organizations  posed  significant  challenges. 

Thus,  given  the  challenges,  the  fact  that  a  Nunn-McCurdy  breach  occurred  is 
not  surprising,  especially  since  many  aspects  of  the  program  were  influenced  by  forces 
and  factors  outside  the  Department  of  Defense  (DoD).  That  fact  notwithstanding, 
the  breach  occurred  and,  by  law,  DoD  had  to  respond.  The  next  chapter  of  this  report 
provides  the  root  cause  analysis  as  defined  in  the  Weapon  Systems  Acquisition  Reform 
Act  (WSARA)  legislation.2 

The  second  topic  dealt  with  in  the  report  pertains  to  the  management  of  acquisi¬ 
tion  programs  in  DoD.  The  high  cost  of  acquiring  major  items  of  equipment  and  the 
cost  overruns  that  have  occurred  have  prompted  acquisition  officials  to  ask  for  a  way 
to  analyze  the  progress  of  acquisition  portfolios,  that  is,  portfolios  that  include  several 
individual  acquisition  programs,  such  as  helicopters  or  satellites.  Such  a  methodology 
would  enable  acquisition  officials  to  monitor  such  factors  as  cost  growth.  Identifying 
cost  growth  across  a  suite  of  programs  might  enable  those  involved  in  acquisition  to 
take  preemptive  steps  and  avoid  cost  overruns.  It  would  also  enable  them  to  identify 


1  A  weapon  system  acquisition  program  incurs  a  unit  cost  growth  breach  when  the  unit  cost  reaches  or  exceeds 
limits  or  thresholds  specified  in  federal  law.  Such  cost  growth  breaches  are  commonly  known  as  Nunn-McCurdy 
breaches,  in  reference  to  the  original  Nunn-McCurdy  amendment,  although  this  term  does  not  appear  in  federal 
law. 

2  Public  Law  111-23,  Weapon  Systems  Acquisition  Reform  Act  of 2009,  May  22,  2009. 
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programs  that  are  performing  particularly  well,  thus  generating  lessons  that  might  be 
applied  elsewhere. 


Organization  of  the  Report 

The  report  contains  two  substantive  parts,  each  with  a  number  of  chapters.  Part  One 
provides  the  root  cause  analysis  of  JPALS,  and  Part  Two  describes  the  portfolio  meth¬ 
odology.  The  report  has  two  appendixes.  Appendix  A  pertains  to  the  JPALS  program, 
and  Appendix  B  pertains  to  the  portfolio  methodology. 


Part  One 

Joint  Precision  Approach  and  Landing  System  Increment  1A 


CHAPTER  TWO 


Joint  Precision  Approach  and  Landing  System  Increment  1A 


In  January  2014,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (USD  [AT&L])  was  informed  by  the  Navy  that  the  JPALS  Inc.  1A  acqui¬ 
sition  program  was  going  to  have  a  critical  Nunn-McCurdy  unit  cost  breach.1  This 
event  triggered  a  statutory  process  mandated  in  the  2009  WSARA  legislation  to  per¬ 
form  a  root  cause  analysis  on  major  defense  acquisition  programs  (MDAPs)  whose  cost 
growth  exceeds  the  threshold  as  detailed  in  10  U.S.  Code  §2433  on  Unit  Cost  Report¬ 
ing.  The  JPALS  Inc.  1A  major  weapon  system  acquisition  program  is  “a  Global  Posi¬ 
tioning  System  (GPS)-based  precision  approach  and  landing  system  that  will  replace 
several  aging  and  obsolete  aircraft  landing  systems  with  a  family  of  systems  that  is 
more  affordable,  will  function  in  more  operational  environments,  and  support  all  DoD 
Land  and  Sea  Based  applications.”2  This  part  of  the  report  describes  the  JPALS  Inc. 
1A  root  cause  analysis,  including  program  background,  circumstances  that  led  to  the 
breach,  and  findings  and  future  considerations.  This  is  not  intended  to  be  a  complete 
JPALS  Inc.  1A  program  history,  and  so  it  does  not  attempt  to  deal  with  every  element 
of  the  program.3  Rather,  we  have  attempted  to  identify  aspects  of  the  program  that  are 
relevant  to  the  explanation  of  the  Nunn-McCurdy  unit  cost  breaches. 


Research  Approach 

The  information  used  in  this  analysis  was  mostly  drawn  from  official  primary  source 
documentation.  We  reviewed  a  wide  range  of  documentary  evidence  including  Acqui¬ 
sition  Decision  Memoranda  (ADMs);  Acquisition  Strategies;  Analysis  of  Alternatives 


1  CAPT  Darrell  D.  Lack,  U.S.  Navy,  “Program  Deviation  Report  Addendum  for  the  Joint  Precision  Approach 
and  Landing  System  Increment  1A  Program,”  memorandum  for  the  Under  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics),  Department  of  the  Navy,  Program  Executive  Office,  Tactical  Aircraft  Programs, 
January  28,  2014. 

2  Department  of  Defense,  Joint  Precision  Approach  and  Landing  System  Increment  1 A  (JPALS  Inc.  1A),  Selected 
Acquisition  Report,  December  31,  2012b,  p.  5. 

3  We  have  provided  a  table  of  key  program  events  in  Appendix  A  that  provides  additional  program  history. 
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(AoA);  Acquisition  Program  Baseline  (APB);  Defense  Acquisition  Executive  Summary 
(DAES);  Selected  Acquisition  Reports  (SARs);  Army,  Navy,  and  Air  Force  budget 
materials;  cost  analyses  from  multiple  sources;  and  cumulative  earned  value  man¬ 
agement  system  data  on  the  major  JPALS  Inc.  1A  contracts.  Other  key  information 
sources  included  Letters  of  Notification  to  Congress  of  the  Nunn-McCurdy  Breach, 
Nunn-McCurdy  Overarching  Integrated  Product  Team  cost  and  management  brief¬ 
ings,  and  Program  Deviation  Reports  (PDRs).  We  also  reviewed  the  Federal  Aviation 
Administration  (FAA)  Federal  Radionavigation  Plans  given  the  connection  between  the 
FAA  and  DoD  in  this  acquisition  program,  and  the  Navy’s  Precision  Approach  Land¬ 
ing  Capability  Roadmap.  In  addition,  we  discussed  the  JPALS  Inc.  1A  with  personnel 
in  the  program  office,  with  service-level  staffs,  with  the  prime  contractor,  and  with  staff 
in  the  Office  of  the  Secretary  of  Defense  (OSD).  Finally,  we  thoroughly  searched  the 
trade  literature  and  Government  Accountability  Office  (GAO)  audits  of  the  program. 
Sources  used  in  this  root  cause  analysis  appear  in  the  list  of  references  at  the  end  of  the 
report.  This  report  was  vetted  by,  and  approved  for  public  release  by,  the  Office  of  the 
Under  Secretary  of  Defense  (OUSD)  (AT&L),  the  Performance  Assessments  and  Root 
Cause  Analyses  (PARCA)  office,  the  Naval  Air  Traffic  Management  Systems  Program 
Office  (PMA213),  and  senior  officials. 

Program  Cost  Growth  and  Certification  Requirements 

Unit  cost  growth  breaches  are  computed  using  definitions,  formulas,  and  thresholds 
provided  in  federal  law.  10  U.S.  Code  (USC)  §2432  and  §2433  provide  “program 
acquisition  unit  cost”  and  “procurement  unit  cost,”  formulas  for  computing  unit  cost 
growth  and  thresholds  for  incurring  significant  and  critical  cost  growth  breaches. 
Table  2.1  summarizes  the  unit  cost  threshold  definitions  in  federal  law. 


Table  2.1 

Breach  Thresholds 


Level 

Unit  Cost 

Current 

Baseline 

Original 

Baseline 

Significant 

PAUC 

>15% 

>30% 

APUC 

>15% 

>30% 

Critical 

PAUC 

>25% 

>50% 

APUC 

>25% 

>50% 
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Organization  of  Part  One 

Chapter  Three  provides  an  overview  of  the  JPALS  Increment  1A  program.  It  describes 
the  motivation  for  the  program,  the  key  decisions  made  between  1997  and  2005,  and 
the  overall  goals  for  the  program.  Chapter  Four  then  discusses  the  Milestone  B  pro¬ 
gram  and  the  framing  assumptions  that  underpinned  the  program.  Chapter  Five  con¬ 
tains  the  root  cause  analysis  of  the  program,  and  it  points  out  some  of  the  limitation  of 
that  analysis.  Finally,  Chapter  Six  provides  our  conclusions. 


CHAPTER  THREE 


JPALS  Increment  1A  Program  Overview 


Defining  the  Need  for  a  Precision  Approach  Landing  System 

JPALS  was  envisioned  as  a  new,  all-weather  precision  landing  system  to  be  used  by  the 
Air  Force,  Army,  and  Navy.1  Following  a  recurring  problem  of  delays  in  the  Bosnia 
airlift  missions  in  the  1990s,  the  Assistant  Secretary  of  Defense  for  Command,  Con¬ 
trol,  Communications  and  Intelligence  directed  a  study  to  analyze  existing  landing 
technology.  The  Air  Force  asked  the  DoD  Policy  Board  on  Federal  Aviation  to  study 
and  produce  a  Mission  Needs  Statement  (MNS)  that  would  then  be  validated  by  the 
Joint  Requirements  Oversight  Council  (JROC).2  The  MNS  was  approved  in  August 
1995  and  indicated  that  “a  need  exists  to  provide  a  rapidly  deployable,  adverse  weather, 
adverse  terrain,  day-night,  survivable,  and  mobile  precision  approach  and  landing 
capability  (PALC)  that  supports  the  principles  of  forward  presence,  crisis  response  and 
mobility.”3  This  response  reflected  a  broader  perception  that  the  U.S.  military’s  post- 
Cold  War  role  would  evolve  into  that  of  the  world’s  peacekeeper  and  would  require 
working  with  and  landing  on  commercial  and  foreign  military  fields,  whose  technol¬ 
ogy  systems  may  not  be  compatible  with  U.S.  landing  technology.4  The  JPALS  systems 
had  the  following  three  main  priorities: 

•  Give  U.S.  forces  the  ability  to  land  on  an  aircraft  carrier’s  deck  or  a  primitive  air¬ 
strip  in  bad  weather  when  visibility  is  low. 

•  Provide  accurate  information,  even  in  the  face  of  enemy  attempts  to  jam  the  land¬ 
ing  signal,  falsify  landing  data,  or  destroy  ground  devices. 


1  “Air  Force  Prepares  for  May  30  DAB  on  $1  Billion  Joint  Landing  System,”  Inside  the  Air  Force,  April  26,  1996. 

2  U.S.  Air  Force,  Director,  Operational  Test  and  Evaluation,  “FY97  Annual  Report  on  Joint  Precision  Approach 
and  Landing  System,”  Global  Security,  undated. 

3  “Final  Mission  Need  Statement  USAF  002-94  Joint  USAF-USN  Mission  Need  Statement  for  Precision 
Approach  and  Landing  Capability,”  August  8,  1994. 

4  “JPALS  Gives  Pilots  Autonomous  Landing  Capability  in  Inclement  Weather,”  Inside  the  Air  Force,  July  26, 
1996. 
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•  Work  with  all  Air  Force,  Navy,  and  Army  platforms,  as  well  as  with  commercial 
and  allied  aircraft.5 

Taking  the  lead,  the  Air  Force  initially  considered  a  number  of  systems  and  com¬ 
binations  of  systems,  some  still  developmental,  to  satisfy  these  requirements  and  still 
be  implemented  internationally.  In  the  international  and  commercial  spheres,  several 
major  systems  were  scheduled  to  be  phased  out,  and  there  was  no  consensus  on  what 
type  of  technology  would  be  commercially  viable.  Many  U.S.  and  European  airfields 
used  the  ILS,  which  at  the  time  was  also  the  Air  Force’s  primary  landing  system. 
However,  there  was  discussion  of  possibly  reducing  the  reliance  on  ILS  as  GPS-based 
precision  approach  systems  matured.  The  FAA  announced  plans  to  end  its  use  of  the 
Microwave  Landing  System  and  considered  pursuing  the  Wide  Area  Augmentation 
System  (WAAS)  and  potentially  the  Local  Area  Augmentation  System  (LAAS).6  How¬ 
ever,  GPS  can  be  jammed,  and  pilots  may  not  realize  that  their  location  is  wrong  if  a 
malfunction  occurs  in  bad  weather.7  Several  interoperable  PALC  systems  were  identi¬ 
fied  in  the  LAA’s  Federal  Radionavigation  Plans  (LRPs)  that  needed  to  be  considered 
during  planning  throughout  the  JPALS  life  cycle: 

The  current  worldwide  standard  system  for  precision  approach  and  landing  is 
the  ILS.8  Ground-Based  Augmentation  System  (GBAS) 9  will  provide  precision 
approach  capability  in  the  future.  The  WAAS  SBAS10  technically  does  not  provide 
a  precision  approach  capability,  but  provides  service  that  is  functionally  equivalent 
to  a  Category  I  (CAT  I)  ILS11  approach  at  airports  with  the  appropriate  infrastruc- 


5  “New  Military  Landing  System  Expected  to  Rely  on  Commercial  Technology,”  Inside  the  Air  Force,  August  2, 
1996. 

6  “Air  Force  Prepares  for  May  30  DAB  on  $1  Billion  Joint  Landing  System,”  1996. 

7  “JPALS  Gives  Pilots  Autonomous  Landing  Capability  in  Inclement  Weather,”  1996. 

8  According  to  the  FAA,  ILS  is  the  predominant  system  supporting  precision  approaches  in  the  United  States. 
With  the  advent  of  GPS-based  precision  approach  systems,  the  role  of  ILS  will  be  reduced.  ILS  may  continue  to 
be  used  to  provide  precision  approach  service  at  major  terminals. 

9  According  to  the  FAA,  the  LAAS  is  a  ground-based  GPS  augmentation  system  being  developed  by  the  FAA. 
It  is  expected  to  provide  the  required  accuracy,  integrity,  and  availability  for  Category  II  and  III  (categories  are 
explained  in  the  footnote  below)  precision  approaches  and  to  increase  the  availability  of  Category  I  services. 
LAAS  may  be  used  to  support  parallel  runway  operations,  runway  incursion  warnings,  high-speed  turnoffs, 
missed  approaches,  departures,  vertical  takeoffs,  and  surface  operations.  LAAS  will  also  support  area  navigation 
(RNAV)  operations. 

10  According  to  the  FAA,  the  WAAS  is  a  satellite-based  GPS  augmentation  system  being  developed  by  the  FAA. 
It  is  expected  to  provide  lateral  and  vertical  navigation  for  all  phases  of  flight  in  the  National  Airspace  System 
(NAS)  except  Category  II  and  III  precision  approaches. 

11  From  the  Federal  Aviation  Administration,  Aeronautical  Information  Manual,  April  2014: 

The  lowest  authorized  ILS  minimums,  with  all  required  ground  and  airborne  systems  components  operative,  are: 
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ture.  LPV  can  provide  approach  capability  as  low  as  a  200  ft  decision  altitude  and 

V2  mi  visibility  minimum  similar  to  the  lowest  CAT  I  minimums.12 

The  North  Atlantic  Treaty  Organization  (NATO),  the  Navy,  and  the  Army  pri¬ 
marily  used  the  Precision  Approach  Radar  (PAR)  system,  which  was  being  phased  out 
because  of  its  manpower-intensive  operation  and  maintenance  requirements  and  its 
lack  of  civilian  interoperability. 

JPALS  was  incorporated  into  the  Air  Mobility  Command’s  1997  master  plan, 
which  indicated  its  goal  to  “embark  on  a  10-  to  15-year  effort  to  develop,  procure,  inte¬ 
grate,  and  install  a  replacement  for  ILS/PAR  on  the  ground  and  in  more  than  10,000 
aircraft.”13  The  total  cost  of  JPALS  was  expected  to  be  between  $1.8  and  $3.5  billion 
and  would  be  shared  among  the  services.14  The  Air  Force  led  a  joint  military  and  FAA 
working  group  to  study  possible  precision  approach  and  landing  systems,  eventually 
concluding  that  no  single  electronic  package  could  meet  all  the  needs  identified  by 
the  JPALS  team.15  It  was  eventually  determined  that  not  one,  but  two  hybrid  solu¬ 
tions  would  be  required  because  of  DoD’s  opposition  to  the  idea  of  equal  military  and 
civilian  access  to  the  GPS  signal.  This  was  a  setback  to  the  “joint”  part  of  JPALS  and 
had  implications  for  the  acquisition  costs  and  ability  to  interoperate  with  civilian  and 
military  systems.16 

As  part  of  the  Phase  0  “Concept  Exploration,”  the  Air  Force  approved  an  AoA 
in  September  1997  for  JPALS,  which  selected  seven  technologies  to  consider.  Three  of 
these  technologies  were  selected  for  review  in  the  risk-reduction  phase.  The  Local  Area 
Differential  Global  Positioning  System  was  best  suited  for  fixed-base  and  special  mis- 


(a)  Category  I.  Decision  Height  (DH)  200  feet  and  Runway  Visual  Range  (RVR)  2,400  feet  (with  touchdown 
zone  and  centerline  lighting,  RVR  1,800  feet),  or  (with  Autopilot  or  Flight  Director  (FD)  or  Head  up  Display 
(HUD),  RVR  1,800  feet); 

(b)  Special  Authorization  Category  I.  DH  150  feet  and  Runway  Visual  Range  (RVR)  1,400  feet,  HUD  to  DH; 

(c)  Category  II.  DH  100  feet  and  RVR  1,200  feet  (with  autoland  or  HUD  to  touchdown  and  noted  on  authoriza¬ 
tion,  RVR  1,000  feet); 

(d)  Special  Authorization  Category  II  with  Reduced  Lighting.  DH  100  feet  and  RVR  1,200  feet  with  autoland 
or  HUD  to  touchdown  and  noted  on  authorization  (touchdown  zone,  centerline  lighting,  and  Approach  Lighting 
System  with  Sequence  Flashing  Lights  (ALSF-2)are  not  required); 

(e)  Category  Ilia.  No  DH  or  DH  below  100  feet  and  RVR  not  less  than  700  feet; 

(f)  Category  Illb.  No  DH  or  DH  below  50  feet  and  RVR  less  than  700  feet  but  not  less  than  150  feet;  and 

(g)  Category  IIIc.  No  DH  and  no  RVR  limitation. 

12  Department  of  Defense,  Department  of  Homeland  Security,  and  Department  of  Transportation,  Federal 
Radionavigation  Plan,  Washington,  D.C.,  2012,  pp.  4-12  and  4-13. 

^  “AMC  Plan  Highlights  Need  for  Near-Term  Austere  Field  Landing  System,”  Inside  the  Air  Force,  November  8, 
1996. 

^  “JPALS  Needs  Additional  $12  Million  for  DEM/VAL,  According  to  PBD,”  1996. 

^  “JPALS  on  Tight  Schedule  for  Improving  Aircraft  Approach  and  Landing,”  1997. 

16  “jpALS  Effort  Will  Yield  Separate  Solutions  for  Military  and  Civilian  Users,”  Inside  the  Air  Force,  September 
26,  1997. 
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sion  environments;  the  upgraded  Automatic  Carrier  Landing  System  (ACLS)  worked 
best  for  shipboard  environments;  and  the  Precision  Approach  Radar  and  Hybrid 
Instrument  Landing  System  (PAR/ILS)  was  best  suited  for  tactical  backup  purposes.17 
Out  of  these,  the  Local  Area  Differential  Global  Positioning  System  was  designated  the 
“primary  alternative”;  however,  because  JPALS  will  result  in  a  family  of  systems,  other 
technologies,  such  as  the  LAAS,  which  was  being  pursued  by  the  FAA,  were  being 
closely  watched  as  potential  candidates  for  secondary  systems.18 

In  May  1998,  after  the  AoA  was  complete,  the  JPALS  program  was  set  to  submit 
an  Operational  Requirements  Document  (ORD)  for  approval  as  well  as  to  reach  Mile¬ 
stone  I  and  enter  the  next  program  phase  of  definition  and  risk  reduction.  However, 
the  JROC  review  and  decision  was  cancelled  because  of  lack  of  funding,  which  left  the 
program  in  the  concept  development  phase.19 

JPALS  Decision  Landscape  (1997  and  2005) 

The  JPALS  effort  had  two  AoAs:  in  1997  and  2005.  As  the  funding  constraint  abated 
in  2004,  the  USD  (AT&L)  directed  that  the  original  AoA  from  1997  be  updated 
to  reflect  the  effect  of  technological  advances,  new  capabilities,  and  improvements  to 
other  alternatives  since  the  original  analysis.  The  changing  conditions  involved  the 
FAA’s  plans  for  a  ground-based  GPS-augmentation  system,  the  use  of  GPS  navigation 
as  an  international  standard,  changes  to  aircraft  avionics  in  DoD,  and  the  increasing 
use  of  Unmanned  Aircraft  Systems  (UAS).  The  outcome  of  the  revised  AoA  shaped 
the  JPALS  program  path  going  forward.  The  result  was  a  joint  program  that  needed 
to  consider  that  Air  Force  and  Army  needs  were  currently  being  met  by  ILS  and  PAR, 
and  that  the  Navy  had  capability  gaps  involving  UAS  and  auto-land  using  GPS.  In 
addition  to  DoD  needs,  interoperability  with  civil  systems  was  considered.  In  late 
2005,  with  the  implementation  of  the  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  process,  the  Joint  Staff  directed  that  the  MNS  be  converted  to  an  Ini¬ 
tial  Capabilities  Document  (ICD),  which  was  approved,  and  the  updated  AoA  was  val¬ 
idated  by  the  Air  Force  Requirements  for  Operational  Capability  Council  (AFROCC). 


17  “Services  Seek  Assistance  in  Planning  New  Precision  Approach  System,”  Inside  the  Air  Force,  December  19, 

1997. 

18  “JPALS  to  Focus  on  Reducing  Risks,  Vulnerability  of  Local  Differential  GPS,”  Inside  the  Air  Force,  March  13, 

1998. 

19  “Making  Waves,”  Inside  the  Navy,  May  18,  1998. 
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JPALS  Goal:  Interoperability  Between  DoD,  FAA,  and  NATO  for 
Precision  Approach  and  Landing  Using  GPS-Based  Technology 

From  the  beginning  of  the  JPALS  effort,  the  Department  of  Transportation  (DOT) 
(FAA)  was  involved  because  under  49  USC  §101,  DOT  is  responsible  for  safe  and  effi¬ 
cient  transportation.  The  FAA  is  also  required  to  develop  a  combined  civil  and  military 
aviation  system.20  Additionally,  the  FAA  must  “select  procedures,  facilities,  and  devices 
that  will  best  serve  those  needs  and  promote  maximum  coordination  of  air  traffic  con¬ 
trol  and  air  defense  systems.”21  Although  FAA  did  not  fund  the  JPALS  program,  its 
decisions  regarding  PALC  were  considered  in  DoD  planning. 


Figure  3.1 

JPALS  Interoperability 
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20  Public  Law  85-726,  72  Stat.  737,  Federal  Aviation  Act  of  1958  (Ref.  16). 

21  Department  of  Defense,  Department  of  Homeland  Security,  and  Department  of  Transportation,  Federal 
Radionavigation  Plan ,  2012,  p.  1-8. 
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One  main  goal  behind  JPALS  was  to  create  interoperability  in  precision  approach 
and  landing  using  GPS -based  technology  within  DoD  and  also  with  the  FAA  and 
NATO.  “Civil  interoperability  is  a  ‘Key  Performance  Parameter’  to  this  DoD  system 
[JPALS].  Funding  and  implementation  of  the  JPALS  system  will  be  primarily  depen¬ 
dent  on  moving  forward  with  the  FAA’s  GBAS  program.”22  Figure  3.1  lists  the  various 
systems  that  DoD  needed  to  take  into  account  when  designing  and  producing  JPALS. 
Within  this  figure  can  also  be  seen  the  wide  variety  of  technologies  that  existed  in  the 
services,  FAA,  and  NATO  regarding  precision  approach  and  landing. 


22  Federal  Aviation  Administration,  National  Airspace  System  Capital  Investment  Plan  FY2013—2017,  2011,  p.  75. 
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Following  the  AoA  approval,  the  JPALS  Capability  Development  Document  (CDD) 
was  approved  by  the  JROC  (only  Increments  1  and  2  of  seven  were  approved)  in  March 
2007.  Given  the  sea-based  nature  of  the  first  two  increments,  the  JROC  directed  the 
Navy  to  take  over  as  lead  service  from  the  Air  Force.  As  the  program  approached 
Milestone  B,  an  Acquisition  Strategy  was  approved  by  USD  (AT&L)  in  July  2007. 
The  approved  JPALS  Acquisition  Strategy  laid  out  a  plan  that  consisted  of  seven  incre¬ 
ments,  based  on  technology  maturity  and  service  needs.  The  decision  was  then  made 
to  separate  the  first  increment  into  two  sea-based  phases  (A  and  B).  The  JPALS  Inc.  1A 
SAR  from  December  2009  provides  the  full  listing  of  increments  and  their  descrip¬ 
tion  (see  Table  4.1).  It  is  important  to  note  that  only  the  requirements  for  Increments  1 

Table  4.1 

JPALS  Increments 


Increment 

Name  Description 


1A  Sea-based,  ship-based  systems  (Navy) 

IB  Sea-based  aircraft  integration 

2  Encompasses  all  fixed  and  mobile  systems  that  support  200  feet  DH  and  Vi  statute  mile  (SM) 
visibility  that  supports  auto-land  for  properly  equipped  aircraft  (to  be  executed  by  the  Air 
Force) 

3  Encompasses  fixed  and  mobile  systems  to  support  FAA  certification  to  100  feet  DH  and  %  SM 
visibility  and  a  sea-based  system  that  supports  auto-land  for  properly  equipped  aircraft 

4  Will  provide  a  sea-based  JPALS  capability  that  supports  100  feet  DH  and  'A  nautical  mile 
(NM)  visibility,  including  auto-land  and  unmanned  aerial  vehicle  (UAV)  support 

5  Will  encompass  land-based  man-pack  systems  certified  to  minimums  based  on  service  needs 

6  Will  support  Special  Operations  Forces,  mobility  missions,  and  subsequent  combat 
operations  with  an  autonomous  approach  and  landing  capability 

7  Is  an  upgrade  to  the  sea-based  backup  capability,  involving  reliability,  maintainability,  and 
life  cycle  improvements  to  the  AN/SPN-41  Instrument  Carrier  Landing  System 

SOURCE:  Department  of  Defense,  Joint  Precision  Approach  and  Landing  System  Increment  1A  (JPALS  Inc. 

1A),  Selected  Acquisition  Report,  December  2009a,  p.  5. 
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and  2  were  approved  by  the  JROC,  and  the  program  plan  that  was  approved  for  Mile¬ 
stone  B  was  only  Increment  1A  of  Increment  1. 

Increment  1A  had  a  Milestone  B  approval  by  the  USD  (AT&L)  in  June  2008. 
At  the  same  time,  it  was  designated  an  Acquisition  Category  (ACAT)  ID  acquisition 
program.1  The  System  Development  and  Demonstration  (SDD)  contract  was  com¬ 
petitively  awarded  to  Raytheon  Corporation.  The  losing  team  lodged  a  formal  bid 
protest  with  the  GAO,  but  it  was  withdrawn  three  months  later,  so  the  JPALS  Inc. 
1A  program  was  minimally  affected  by  the  protest.  An  APB  was  formally  approved  in 
December  2008.  As  reported  in  the  2009  SAR,  the  mission  for  the  approved  JPALS 
program  is  as  follows: 

Joint  Precision  Approach  and  Landing  System  (JPALS)  is  a  program  with  Tri-Ser- 
vice  partners  for  acquisition  of  JPALS  including  the  Navy  .  .  .  Air  Force  .  .  .  and 
Army .  .  .  JPALS  is  a  Global  Positioning  System  (GPS)-based  precision  approach 
and  landing  system  that  will  replace  several  aging  and  obsolete  unique  aircraft 
landing  systems.  JPALS  will  provide  a  family  of  systems  that  is  more  affordable, 
will  function  in  more  operational  environments,  and  will  support  all. .  .DoD  Land 
and  Sea  Based  applications.  The  National  Defense  Strategy  of  the  United  States  of 
America  calls  for  highly  mobile  forces  that  can  rapidly  respond  to  crises  worldwide. 
Success  in  meeting  this  challenge  requires  the  ability  to  land  aviation  assets  vir¬ 
tually  anywhere,  at  any  time.  JPALS  will  provide  this  capability  by  being  rapidly 
deployable,  survivable  and  interoperable  among  the  U.S.  Services  and  with  U.S. 
allies,  as  well  as  with  civil  aircraft  and  landing  facilities.  JPALS  will  eventually  sup¬ 
port  unmanned  and  highly  automated  aircraft,  and  will  be  able  to  operate  during 
restricted  Emission  Control  (EMCON)  conditions.2 

Following  Milestone  B,  the  Increment  1A  program  completed  a  series  of  key 
events  before  the  Nunn-McCurdy  breach  in  2014.  These  included: 

•  System  Requirements  Review  (SRR)-2  (January  2009) 

•  Integrated  Baseline  Review  (IBR)  (April  2009) 

•  System  Functional  Review  (SFR)  (June  2009) 

•  Preliminary  Design  Review  (PDR)  (December  2009) 

•  Critical  Design  Review  (CDR)  (December  2010) 


1  ACAT  I  programs  are  MDAPs.  An  MDAP  is  a  program  that  is  not  a  highly  sensitive  classified  program  and 
is  designated  by  USD  (AT&L)  as  an  MDAP;  or  that  is  estimated  to  require  eventual  expenditure  for  research, 
development,  test,  and  evaluation  (RDT&E),  including  all  planned  increments,  of  more  than  $480  million  (fiscal 
year  [FY]  2014  constant  dollars)  or  procurement,  including  all  planned  increments,  of  more  than  $2.79  billion 
(FY  2014  constant  dollars).  ACAT  I  programs  have  two  subcategories:  ACAT  ID  for  which  the  Milestone  Deci¬ 
sion  Authority  (MDA)  is  USD  (AT&L).  The  “D”  refers  to  the  Defense  Acquisition  Board  (DAB),  which  advises 
USD  (AT&L)  at  major  decision  points. 

2  Department  of  Defense,  2009a,  p.  5. 
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•  Delivery  of  three  engineering  development  models  (EDMs)  to  the  contractor 
system  integration  lab  (2010) 

•  Completed  early  testing  of  the  GPS  receivers  onboard  Landing  Helicopter  Deck 
(LHD-1),  which  mitigated  several  program  risks  before  the  beginning  of  formal 
developmental  test  (July  2011) 

•  Test  Readiness  Review  (TRR)  (May  2012) 

•  Commenced  Integrated  Test  (June  2012). 


JPALS  Increment  1A  Program  Cost  and  Planned  Quantity  After 
Milestone  B 

The  JPALS  program’s  2008  to  2013  SARs  informed  our  understanding  of  program  cost 
and  planned  quantity  for  the  JPALS  Inc.  1A  program.  The  acquisition  program  that 
was  defined  at  Milestone  B  is  only  Increment  1A.  Subsequent  increments  would  have 
their  own  Milestone  B  as  they  are  introduced.  At  the  macro  level  shown  in  Ligure  4.1, 
the  JPALs  Inc.  1A  program  has  almost  constant  cost  and  quantity  from  2008  to  2012. 
The  majority  of  the  program’s  planned  spending  falls  under  RDT&E.  Procurement 
costs  for  this  program  were  roughly  20  percent  of  estimated  total  program  costs.  The 
2013  SAR  reported  the  reorganization  of  the  program.  On  the  financial  side,  the  reor¬ 
ganization  involved  shifting  funds  from  future  increments  into  the  current  Increment 

Figure  4.1 

JPALS  Program  Cost  and  Planned  Quantity  Since  Milestone  B,  2008  Through  2013 
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1A  program  and  reporting  in  the  SAR.  As  for  quantity,  there  was  a  change  in  the 
requirement  for  shore-based  training  units  prompting  the  decrease  in  2013  as  the  Air 
Force  and  Army  assumed  less  active  roles  in  the  program.  Other  details  of  the  reorga¬ 
nization  are  discussed  further  in  Chapter  Five. 

As  quantity  remained  consistent  between  2008  and  2012,  the  unit  cost  of  the 
JPALS  program  did  not  change  significantly  until  the  reduction  in  quantity  reported 
in  the  2013  SAR  (see  Figure  4.2).  Unit  costs  are  measured  with  two  congressionally 
mandated  metrics:  program  acquisition  unit  cost  (PAUC)  and  average  procurement 
unit  cost  (APUC).3  The  majority  of  costs  are  within  RDT&E,  which  is  reflected  in 
the  PAUC.  With  the  restructure  in  2013,  both  PAUC  and  APUC  exceeded  the  Nunn- 
McCurdy  thresholds  for  cost  growth. 


JPALS  Increment  1A  Framing  Assumptions 

As  part  of  our  analysis  of  the  JPALS  Inc.  1A  program,  we  collected  available  fram¬ 
ing  assumptions  for  the  program  and  supplemented  them  with  additional  information 
as  necessary.  Framing  assumptions  are  the  foundational  assumptions  that  underlie  a 
program’s  structure  and  acquisition  strategy  and  determine  why  critical  elements  of 

Figure  4.2 

JPALS  Unit  Cost  and  Planned  Quantity  Since  Milestone  B,  2008  Through  2013 
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3  PAUC  is  calculated  by  dividing  the  program  acquisition  cost  by  the  total  quantity.  APUC  includes  only  pro¬ 
curement  cost  divided  by  procurement  quantity. 
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a  program’s  acquisition  strategy  were  designed  (planned)  the  way  they  were.  All  pro¬ 
grams  have  framing  assumptions,  explicitly  or  implicitly,  at  program  initiation.  If  these 
assumptions  turn  out  to  be  incorrect,  program  execution  can  be  adversely  affected.4 

As  a  program  planning  and  management  tool,  framing  assumptions  should  be 
generated  at  the  beginning  of  a  program  and  then  monitored  during  program  execu¬ 
tion  to  uncover  significant  deviations  that  may  adversely  affect  cost,  schedule,  and  per¬ 
formance  outcomes.  However,  framing  assumptions  can  also  be  used  as  part  of  a  root 
cause  analysis;  the  exercise  enables  analysts  to  identify  a  small  number  of  key  assump¬ 
tions  that  drive  outcomes. 

RAND  identified  the  following  framing  assumptions  for  JPALS: 

•  GPS -based  precision  approach  and  landing  technologies  are  suitable  for  multiple 
ship-  and  shore-based  environments  and  aircraft.  After  a  number  of  studies  and 
limited  demonstration  in  the  early  1990s  (under  Air  Force  lead),  DoD  and  FAA 
settled  on  a  GPS -based  approach  to  meet  multiagency  and  international  require¬ 
ments  for  precision  approach  and  landing  capabilities.  A  part  of  this,  from  the 
DoD  perspective,  is  the  need  for  DoD  aircraft  to  be  interoperable  with  the  preci¬ 
sion  landing  systems  of  civilian  airports.  This  assumption  appears  to  remain  valid. 

•  Incremental  development  and  use  of  commercial  off-the-shelf/government  off- 
the-shelf  (COTS/GOTS)  hardware  and  software  will  lower  risks.  JPALS  Inc.  1A 
is  based  on  existing  hardware.  Most  of  the  technical  risk  in  the  program  revolved 
around  development  of  the  algorithms  (software)  that  improve  the  precision  of 
the  GPS  signal.  Given  the  minimal  cost  and  schedule  growth  associated  with 
technical  challenges,  this  assumption  also  appears  to  remain  valid. 

•  Test  assets  (ships  and  aircraft)  will  be  available  as  planned  for  JPALS  installation, 
test,  and  integration.  This  is  somewhat  unique  to  Navy  ships  because  they  have 
very  limited  quantities  available  at  any  given  time  for  operations.  JPALS  Inc.  1A 
required  that  EDMs  be  installed  on  CVN  and  L-class  (Landing  Platform  Dock 
[LPD],  Landing  Helicopter  Dock  [LHD])  ships  for  development  and  operational 
testing.  Installation  and  test  therefore  requires  that  the  intended  ship  supporting 
the  test  be  at  its  home  port  for  sufficient  time.  However,  the  ships  were  not  avail¬ 
able  for  testing  as  intended,  which  resulted  in  some  schedule  slip  and  a  need  to 
adjust  the  test  program  to  meet  these  realities. 

•  FAA  and  DoD  investment  needs  and  time  frame  for  technology  transition  are 
similar.  In  particular,  the  original  structure  of  the  JPALS  program  (to  include 
at  least  Incs.  1A,  IB,  and  2)  was  based  on  the  assumption  that  the  FAA  would 


4  Mark  V.  Arena,  Irv  Blickstein,  Abby  Doll,  Jeffrey  A.  Drezner,  James  G.  Kallimani,  Jennifer  Kavanagh,  Daniel 
F.  McCaffrey,  Megan  McKernan,  Charles  Nemfakos,  Rena  Rudavsky,  Jerry  M.  Sollinger,  Daniel  Tremblay,  and 
Carolyn  Wong,  Management  Perspectives  Pertaining  to  Root  Cause  Analyses  of  Nunn-McCurdy  Breaches,  Volume  4: 
Program  Manager  Tenure,  Oversight  of  Acquisition  Category  II  Programs,  and  Framing  Assumptions,  Santa  Monica, 
Calif.:  RAND  Corporation,  MG-1171/4-OSD,  2013. 
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begin  transition  from  legacy  ILS  to  GPS-based  systems  in  2010  time  frame  (see 
the  discussion  of  FAA  decisionmaking  below).  This  turned  out  not  to  be  the  case. 

•  The  Air  Force,  Army,  and  Navy  will  remain  committed  to  JPALS  as  a  solution  to 
precision  approach  and  landing  interoperability.  When  the  FAA  kept  ILS  oper¬ 
ational  and  deferred  transition  to  a  GPS-based  technology,  the  need  to  equip 
legacy  DoD  aircraft  with  JPALS  disappeared  or  moved  to  the  out-years.  Most  of 
the  Air  Force  and  Army  fleet  already  had  ILS  and  so  did  not  have  the  same  capa¬ 
bility  gap  that  many  Navy  aircraft  had  without  ILS. 

The  JPALS  framing  assumptions  are  consistent  with  both  the  2005  AoA  and 
the  Navy’s  2012-2013  assessment  of  changing  conditions.  Changes  in  these  assump¬ 
tions  and  changes  in  the  environment  defined  by  these  assumptions  significantly 
affected  JPALS’  decision  environment,  forcing  the  Navy  (and  other  JPALS  stakehold¬ 
ers  in  DoD)  to  revisit  the  program  structure  and  objectives  to  align  better  with  actual 
conditions. 


CHAPTER  FIVE 


Root  Cause  Analysis 


As  mentioned  above,  the  JPALS  Inc.  1 A  program  declared  a  Nunn-McCurdy  unit  cost 
breach  in  January  2014  when  a  program  deviation  report  was  signed  by  the  program 
manager  on  January  28,  endorsed  by  the  Navy  Acquisition  Executive,  and  forwarded 
to  the  MDA  on  March  12,  2014.  The  Secretary  of  the  Navy  notified  Congress  of  the 
breach  on  March  19,  2014. 

Table  5.1  provides  the  details  of  the  breach  in  what  has  been  called  the  “speeding 
ticket”  in  past  Nunn-McCurdy  analyses  performed  by  RAND.  The  speeding  ticket 
shows  that  the  APUC  and  PAUC  exceeded  critical  thresholds  against  both  the  original 
baseline  and  the  current  baseline.  The  APUC  exceeded  by  nearly  129  percent  over  both 
baselines,  and  the  PAUC  exceeded  by  approximately  104  percent  over  both  baselines. 
In  addition,  the  cost  growth  expressed  in  dollar  amounts  over  the  original  or  current 
baseline  was  calculated.  Brief  explanations  of  the  immediate  causes  of  the  unit  cost 
growth,  as  reported  in  the  SAR,  are  also  provided  in  the  table. 

Figure  5.1  lays  out  the  key  events  that  provide  context  within  the  program’s  life 
cycle  and  that  eventually  led  to  the  Nunn-McCurdy  breach.  The  events  in  red  boxes 
are  important  pieces  of  the  explanation  of  what  happened  to  trigger  the  breaches.  The 
decisions  of  both  the  FAA  and  the  services  are  noted  in  the  figure,  particularly  that 
the  timing  of  the  ILS  phasedown  became  uncertain,  which  meant  that  the  ILS  would 
continue  to  be  the  standard. 

Root  Cause:  FAA  Transition  to  GPS-Based  Precision  Approach  and 
Landing  Diverges  from  DoD  JPALS  Planning 

Ultimately  the  FAA  decision  to  continue  using  ILS  instead  of  phasing  it  out  and  phas¬ 
ing  in  GPS-based  precision  approach  and  landing  technology  diverged  from  DoD’s 
plan  to  move  toward  GPS -based  technology.  As  a  result,  the  Army  and  the  Air  Force 
reassessed  their  PALC  plans.  The  root  cause  of  the  breach  lies  in  part  on  multiple  deci¬ 
sionmakers  making  multiple  decisions  regarding  the  PALC  that  was  going  to  be  used 
by  each  entity. 

The  Navy  then  had  to  consider  the  consequences  of  the  Air  Force,  Army,  and 
FAA  continuing  to  rely  on  ILS.  More  specifically,  the  Navy’s  proposed  restructur- 
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Table  5.1 

JPALS  Inc.  1A  Speeding  Ticket 


Cost  Growth  Threshold  Breaches 

Current  - 

Estimate  (Source,  Current 

Baseline  Dec  31, 2013,  Quantity 

Unit  Cost  (FY  SAR)  FY  2008  Baseline3  Baseline  (Dec  2013  Cause  in  Explanation 


Program 

$  millions) 

$  millions 
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SAR) 

Source 

in  Source 

JPALS  Inc. 

APUC 
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Over  current 

APUC 

+$10,466 

Critical 

25 

17 

Reduction  in  total 

As  a  result  of 

1 A 

$8,116 

$18,582 

baseline 

+128.96% 

FY  2008  $M 

planned  procurement 

the  Navy's  PALC 

(Dec  2008 

(Dec  2013 

quantities  resulted  in  a 

Roadmap,  it  was 

APB) 

SAR) 

critical  Nunn-McCurdy 

determined  that 

unit  cost  breach  to  the 
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PAUC 

PAUC 

PAUC 

+$27,146 

Critical 

37 

27 
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shore-based  training 

$26,032 

$53,178 

+104.28% 

FY  2008  $M 

systems  would  be 

(Dec  2008 
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APB) 

APUC 

APUC 

Over  original 

APUC 

+$10,466 

Critical 

25 

17 

$8,116 

$18,582 

baseline 

+128.96% 

FY  2008  $M 

(Dec  2008 

(Dec  2013 

APB) 
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PAUC 

PAUC 

PAUC 

+$27,146 

Critical 

37 

27 

$26,032 

$53,178 

+014.28% 

FY  2008  $M 

(Dec  2008 
APB) 


SOURCE:  Department  of  Defense,  Joint  Precision  Approach  and  Landing  System  Increment  1A  (JPALS  Inc.  1A),  Selected  Acquisition  Report, 
December  2013b. 

NOTE:  The  numbers  in  red  indicate  the  "speeding  ticket"  triggering  root  cause  analysis  by  PARCA. 

3  For  the  JPALS  Inc.  1A  program,  both  the  original  and  current  estimates  are  the  same.  The  program  had  a  Milestone  B  in  2008  in  which  both 
the  current  and  original  estimates  started  as  the  same  estimate  based  on  the  APB  at  Milestone  B.  After  Milestone  B,  there  was  little  cost  growth 
or  other  changes  to  the  program  that  would  have  warranted  a  change  in  the  current  baseline;  therefore,  the  current  and  original  baselines 
remained  the  same  until  the  Nunn-McCurdy  breach. 
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Figure  5.1 

JPALS  Key  Events  Leading  to  a  Nunn-McCurdy  Breach 
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SOURCES:  Department  of  Defense,  JPALS  Inc.  1  SARs,  December  2009  through  December  2013;  Depart¬ 
ment  of  Defense,  Joint  Precision  Approach  and  Landing  System  Acquisition  Strategy  in  Support  of 
Milestone  B  for  JPALS  Increment  1A,  Washington,  D.C.,  June  2008;  Under  Secretary  of  Defense  (A&T), 
ADM,  May  1996;  Department  of  Defense,  Joint  Precision  Approach  and  Landing  System  Updated  Analysis 
of  Alternatives  (AoA)  Study  Report,  November  17,  2005;  Department  of  Defense,  Department  of 
Homeland  Security,  and  Department  of  Transportation,  Federal  Radionavigation  Plans,  2001,  2005,  2008, 
2010,  and  2012;  Federal  Aviation  Administration,  Capital  Investment  Plan  FY  13-17,  March  2012;  Navy, 
Army,  and  Air  Force  budget  documentation;  multiple  Air  Force,  Navy,  and  Army  ADMs. 
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ing  (and  funding  consolidation),  which  was  motivated  by  the  findings  of  the  Navy’s 
PALC  Roadmap  study,  was  initiated  by  a  series  of  “fact-of-life”  changes  to  the  assump¬ 
tions  underpinning  the  original  JPALS  seven-increment  plan.  The  most  fundamental 
of  these  was  the  FAA’s  decision  to  continue  with  ILS  indefinitely  (thus  setting  off  those 
fact-of-life  changes  in  domino  fashion).  Figure  5.2  provides  the  series  and  timing  of  the 
events  discussed  above. 
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Figure  5.2 

Root  Cause  of  the  JPALS  Inc.  1A  Nunn-McCurdy  Breach 


- 1 - 

Beyond  Navy  control 


aAccording  to  a  November  201 2  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ)  memo,  because  of 
recent  affordability  assessment,  ongoing  service  life  extension  on  legacy  systems,  and  a  change  in 
operational  need.  The  Army  did  not  release  an  official  memo,  but  zeroed  out  funding  in  April  2013 
budget  documents  as  did  the  Air  Force.  The  Army  has  a  standing  unfunded  requirement. 
bA  2005  AoA  identified  gaps:  operational  interoperability  with  FAA,  DoD,  and  international  civil  aviation 
organizations;  supportability  in  all  four  operating  environments;  and  an  emerging  gap  in  sea-based 
operations  beginning  in  2013  (CVN21,  Joint  Strike  Fighter,  Joint  Unmanned  Combat  Air  System). 
According  to  multiple  FAA  official  planning  documents  (2001-2012),  the  phasedown  of  the  standard 
precision  and  landing  system,  ILS,  in  favor  of  GPS-based  precision  approach  and  landing  (PAL)  technology 
did  not  begin  as  was  planned  in  2001 .  After  2008,  the  transition  remained  unclear. 
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Evidence  Supporting  Root  Cause  Analysis 

A  cornerstone  of  the  JPALS  implementation  strategy  was  parallel  development  and 
implementation  with  the  FAA  LAAS,  which  was  based  on  the  same  GPS  GBAS  tech¬ 
nology.  However,  since  the  CDD  was  approved,  the  pace  of  the  FAA  LAAS  implemen¬ 
tation  and  phase-down  of  the  legacy  instrument  landing  system  slowed. 

As  part  of  this  analysis,  we  tracked  the  FAA  decisionmaking  regarding  PALC  from 
2001  through  2012  in  its  official  planning  documents.  We  then  focused  on  the  three 
technologies  most  relevant  to  JPALS:  ILS,  WAAS,  and  LAAS.  WAAS  and  LAAS  both 
augment  GPS-precision  approach  and  landing  using  ground-based  stations.  WAAS  is  a 
satellite-based  augmentation  system.  LAAS  is  a  ground-based  augmentation  system.  By 
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2012,  LAAS  had  been  wrapped  into  the  GBAS  category.  Most  important,  we  focused 
on  the  ILS  decisions  because  ILS  is  the  civil  standard  for  precision  approach  and  land¬ 
ing.  The  FAA  was  indecisive  regarding  the  future  of  ILS  because  it  changed  the  ILS 
drawdown  multiple  times.  Table  5.2  provides  information  extracted  from  the  Federal 
Radionavigation  Plans  over  time  for  the  three  key  FAA  PALC  technologies. 

As  explained  above,  the  services  had  to  make  decisions  regarding  JPALS.  The 
Army  budget  reflected  the  final  funding  for  the  Army’s  portion  of  the  program  in  the 
Presidents’  Budget,  2014. 1  The  last  funding  provided  in  the  budget  was  for  FY  2012 
at  $8,297  million.  There  was  no  reported  funding  for  FY  2013  (submitted  in  Febru- 


Table  5.2 

FAA  Decisionmaking  for  Precision  and  Landing  Capability 
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SOURCES:  Department  of  Defense,  Department  of  Homeland  Security,  and  Department  of 
Transportation,  Federal  Radionavigation  Plans,  2001,  2005,  2008,  2010,  2012;  Federal  Aviation 
Administration,  Capital  Investment  Plan  FYs2013-20l7,  March  2012. 


1  Department  of  the  Army,  Exhibit  R-2:  RDT&E  Budget  Item  Justification:  PB  2014  Army,  Washington,  D.C., 
0604201  A:  Aircraft  Avionics,  April  2013. 
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ary  2012)  or  FY  2014.2  In  addition,  an  article  from  Jane’s  Defence  Weekly  indicated 
that  the  Army  would  terminate  JPALS.3  According  to  one  official  associated  with  the 
program,  the  Army’s  reduced  participation  in  JPALS  was  largely  because  of  the  sharp 
drops  in  the  budget.  The  Army  requirement  for  JPALS  remains  but  is  being  carried  as 
an  unfunded  requirement. 

The  Air  Force  budget  said  the  following  about  the  Air  Force’s  transition  out  of  the 
JPALS  program  in  its  April  2013  budget  document: 

As  reflected  in  the  JPALS  CDD,  JPALS  is  intended  to  be  the  next  generation  preci¬ 
sion  approach  and  landing  system  which  would  provide  a  common  interoperable 
system  for  both  DoD  and  civil  use.  A  cornerstone  of  the  JPALS  implementation 
strategy  was  parallel  development  and  implementation  with  the  Federal  Aviation 
Administration  Local  Area  Augmentation  System  (LAAS)  which  was  based  on  the 
same  GPS  Ground  Based  Augmentation  System  (GBAS)  technology.  Flowever, 
since  the  CDD  was  approved,  the  pace  of  the  FAA  LAAS  implementation  and 
phase  down  of  the  legacy  instrument  landing  system  has  slowed.  This,  combined 
with  an  on-going  tech  refresh  (life  extension)  of  the  legacy  Instrument  Landing 
System  (ILS),  development  of  a  Deployable  ILS,  the  fact  that  new  platforms  (F-35, 
KC-46)  will  be  ILS  equipped,  and  the  need  to  fund  higher  Air  Force  priorities, 
the  Air  Force  is  delaying  JPALS  implementation  for  the  foreseeable  future.  The 
Army  has  also  delayed  JPALS  implementation  beyond  the  current  future  years 
defense  plan.  As  JPALS  is  now  early  to  need  for  the  Air  Force,  Land-Based  Incre¬ 
ment  2  development  responsibility  is  being  transferred  from  the  Air  Force  to  the 
Navy.  Both  Service  Acquisition  Executives  are  in  agreement  on  transfer  of  the 
Land-Based  Increment  2.  The  Navy  has  a  near  term  need  in  FY17  for  a  shore  based 
system  to  support  Joint  Strike  Fighter  carrier  landing  training.  The  Air  Force  will 
monitor  the  progress  of  JPALS  implementation  but,  in  the  interim,  continue  the 
use  of  the  legacy  Instrument  Landing  System  (ILS).  ILS  is  supportable  through 
the  2030  timeframe  and,  in  its  fixed  and  deployable  configuration,  will  provide  an 
effective  and  affordable  landing  system  capability.  The  Air  Force  will  also  retain  a 
limited  number  mobile  precision  approach  radars  to  provide  joint  Service  interop¬ 
erability  at  deployed  location  as  all  Navy  and  Army  aircraft  are  not  ILS  equipped.  4 


2  According  to  the  Army  budget  document,  FY  2012  accomplishments  included  the  following:  Completed  the 
Aircraft  Integration  Guide  (AIG)  effort  related  to  the  AH-64D  platform,  Block  III.  Completed  nonrecurring 
engineering  efforts  for  M-Code  development.  Completed  Small  Antenna  System  antijamming  antenna  cosite 
analysis  and  M-Code  recurring  prototyping.  Completed  the  JPALS  Common  Avionics  Technology  Development 
(JCATD)  effort,  and  continued  to  support  JPALS  Increments  1  and  2  development  and  program  management 
coordination  meetings,  technical  interchange  meetings,  and  working  groups. 

3  Daniel  Wasserbly,  “U.S.  Army  Gives  Details  of  the  FY13  Budget  Request  Jane’s  Defence  Weekly,  February  14, 
2012. 

4  Department  of  the  Air  Force,  Exhibit  R-2:  RDT&E  Budget  Item  Justification:  PB  2014  Air  Force,  Washington, 
D.C.,  0603860F  Joint  Precision  Approach  and  Landing  Systems — Dem/Val,  April  2013. 
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As  stated  in  this  quotation,  the  Air  Force  mentioned  that  the  JPALS  Land-Based 
Increment  2  would  be  transferred  to  the  Navy  along  with  the  Increment  2  development 
funding  and  that  the  Air  Force  would  continue  to  use  the  ILS  through  the  2030  time 
frame.  In  addition,  SAF/AQ  provided  a  memorandum  to  the  Assistant  Secretary  of  the 
Navy  for  Research,  Development,  and  Acquisition  on  November  29,  2012,  stating: 
“Due  to  a  recent  affordability  assessment;  an  ongoing  service  life  extension  on  legacy 
landing  systems  and  changes  in  operational  needs,  the  United  States  Air  Force  (USAF) 
requirement  for  JPALS  is  deferred  beyond  the  FYDP;  however,  the  United  States  Navy 
(USN)  need  in  the  FY17  timeframe  remains  intact.”5 

Because  of  the  decisionmaking  environment  of  the  FAA,  the  Army,  and  the  Air 
Force  along  with  DoD  “budget  constraints  and  affordability  concerns,”  the  Navy  per¬ 
formed  an  internal  analysis  of  the  Navy  PALC  requirement.6  The  results  were  pre¬ 
sented  in  June  2013. 7  In  the  January  2014  PDR,  the  Navy  laid  out  more  specifically  a 
path  moving  forward  that  would  lead  to  a  restructured  JPALS  (see  Figure  5.3): 

Under  the  recommended  restructure,  the  JPALS  Ship  System  (currently  desig¬ 
nated  as  Increment  1A)  will  continue  to  be  developed  and  procured  for  use  on 
US  Navy  aircraft  carriers  (CVN-type)  and  amphibious  assault  ships  (LH-type) 
in  support  of  the  F-35B/C  and  Unmanned  Carrier-Launched  Surveillance  and 
Strike  (UCLASS)  programs.  JPALS  will  not  be  integrated  into  legacy  CYN  air¬ 
craft  (prior  JPALS  Increment  IB  aircraft  integration),  nor  will  JPALS  be  developed 
and  procured  for  Navy  and  Marine  Corps  fixed  based  air  stations  or  expedition¬ 
ary  airfields  (prior  JPALS  Increment  2  land-based  capability).  JPALS  will  support 
manned  and  unmanned  auto-land  capability  (prior  JPALS  Increments  3  and  4 
auto-land  and  UAS  support)  and  will  be  incorporated  into  the  current  single  incre¬ 
ment  JPALS  POR.  Separate  from  the  JPALS  program,  civil  Instrument  Landing 
System  (ILS)  capability  will  be  procured  and  deployed  at  Navy  and  Marine  Corps 
fixed-base  air  stations  and  integrated  into  legacy  aircraft  to  address  joint  and  civil 
interoperability  gaps.  Further,  legacy  shipboard  landing  systems  will  be  recapital¬ 
ized  and/or  sustained.8 

The  Navy  also  determined  that  it  would  eliminate  the  shore-based  training  sys¬ 
tems,  which  reflects  the  reduction  in  quantity,  and  Milestone  C  would  slip  three  years. 


5  Lt  Gen  Charles  R.  Davis,  U.S.  Air  Force,  “Joint  Precision  Approach  and  Landing  System  (JPALS)  Increment  2,” 
memorandum  for  the  Assistant  Secretary  of  the  Navy  for  Research,  Development  and  Acquisition,  November 
29,  2012. 

6  Lack,  2014,  p.  1. 

7  CAPT  Brett  K.  Easier,  Department  of  Navy  Precision  Approach  Landing  Capability  (PALC)  Roadmap  to  Lnclude 
Joint  Precision  Approach  Landing  System  (JPALS)  Re-Scope  to  Single  Lncrement  R3B  Brief  Washington,  D.C.,  U.S. 
Navy,  June  10,  2013.  PRE-DECISIONAL/NOT  FOR  RELEASE. 

8  Lack,  2014,  pp.  1-2. 
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Figure  5.3 

JPALS  Restructure:  Which  Platforms  Will  Have  JPALS? 
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In  this  acquisition  program,  the  Navy  has  always  had  a  unique  need  for  GPS- 
based  precision  approach  and  landing  technology,  and  its  decision  to  restructure  JPALS 
was  driven  by  a  unique  confluence  of  factors.  This  unique  need  arises  mainly  because 
various  aircraft  need  to  land  on  ships  and  carriers  with  landing  platforms  that  are 
not  stationary.  The  Navy  has  unique  precision  landing  requirements  where  landing  is 
inherently  more  risky  than  it  is  on  land,  and  those  requirements  could  not  be  fulfilled 
by  means  other  than  JPALS: 

•  sea-based  operations  (e.g.,  landing  on  carrier) 

•  F-35B/C  design  risk/dependency 

•  improved  precision  landing  system  required  for  UCLASS,  which  is  an  aircraft 
with  no  pilot. 

Also,  updating  legacy  Navy  systems  is  more  affordable  than  transitioning  all 
legacy  systems  to  JPALS: 

•  ILS  incorporation  onto  legacy  aircraft  (e.g.,  F/A-18);  solves  interoperability  prob¬ 
lem  within  DoD  and  FAA. 

•  SPN-469  restored  on  CVN  78  and  CVN  79. 


9  AN/SPN-46,  or  ACLS,  is  the  carrier’s  only  fully  automated,  all-weather  approach  landing  aid  for  carrier 
aircraft. 
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•  Supportability  gaps  in  legacy  systems  have  been  addressed. 

•  Potential  for  greatly  improved  UAV  capabilities  in  the  longer  run. 


Contributing  Factor:  Program  Interdependencies  Caused  Cost  Growth 
and  Schedule  Slip  in  Increment  1A 

The  baseline  program — Inc.  1A — incurred  some  modest  cost  and  schedule  growth, 
independent  of  the  factors  affecting  the  Navy’s  decision  to  restructure  the  program. 
Figure  5.4  provides  the  series  of  events  that  resulted  in  cost  growth  in  the  Inc.  1A 
program. 

The  JPALS  Inc.  1A  program  of  record  required  shipboard  Integrated  Testing  and 
Operational  Assessment.  The  testing  was  planned  to  occur  on  the  CVN-77.  According 
to  the  PDR  from  July  2012,  several  shifts  from  2009  to  2012  to  CVN-77  installation 
availability  occurred,  resulting  in  a  delay  in  operational  assessment  (OA)  testing  and, 

Figure  5.4 

The  Effect  of  Program  Interdependencies 


SOURCE:  Lack,  2012a. 

aln  2010,  the  Milestone  C  date  changed  from  February  2013  to  May  2013.  In  April  2012,  Milestone  C  was 
changed  from  May  2013  to  November  2013  resulting  in  an  APB  schedule  breach. 
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consequently,  a  slip  in  Milestone  C  from  May  2013  to  November  2013,  which  was 
an  APB  schedule  breach.10  This  shift  also  resulted  in  a  decrease  of  one  RDT&E  unit 
and  an  increase  of  one  unit  to  procurement.  The  new  procurement  quantity  needed  to 
satisfy  a  new  production  schedule,  which  was  extended  by  three  years.  Consequently, 
some  fixed  cost  increases  are  directly  attributable  to  the  new  production  schedule  to 
include  additional  government  and  contractor  staffing  and  extending  the  production 
contract  by  three  years.  The  increase  to  procurement  units  and  fixed  costs  caused  an 
APB  procurement  cost  breach.* 11 

A  detailed  review  of  SAR  change  explanations  and  DAES  assessments  suggests 
that  three  basic  factors  account  for  approximately  10  percent  of  the  Inc.  1A  program 
cost  growth: 

1.  underestimation  of  required  government  and  contractor  system  engineering, 
system  integration,  and  program  management  personnel  and  activities 

2.  test  asset  availability 

3.  procurement  and  installation  schedule  slip. 

The  underestimation  of  system  engineering  manpower  and  activities  largely 
reflects  the  challenge  of  developing  the  algorithm  that  constitutes  the  core  capability 
of  JPALS.  Test  asset  availability  and  the  schedule  slip  to  accommodate  a  revised  pro¬ 
duction  profile  and  installation  schedule  are  direct  effects  of  program  interdependency. 
The  slip  in  schedule  eventually  caused  an  APB  schedule  breach  to  Milestone  C  and  an 
APB  procurement  cost  breach. 


Findings 

Root  causes  are  an  input  to  understanding  changes  in  program  costs.  We  linked  the 
root  causes  back  to  the  cost  drivers  as  seen  in  program  documentation  and  in  the 
SARs.  We  used  the  rubric  and  color  coding  in  Figure  5.5,  to  help  highlight  where  root 
causes  are  linked  to  technical  factors  that  affected  cost  drivers.  For  instance,  the  FAA 
decisions  led  to  a  restructuring  of  the  requirements,  which  led  to  increased  scope  in 
the  current  SAR.  The  other  cost  drivers  we  identified  include  reduction  in  quantity, 
contract  cost  growth,  and  stretched  schedule. 

Once  we  selected  the  most  appropriate  connections  between  the  root  causes 
and  the  cost  drivers,  data  on  cost  drivers  from  each  of  the  annual  SARs  were  aggre¬ 
gated,  coded  to  the  categories  shown  in  Figure  5.5,  and  analyzed  to  understand  cost 


10  An  APB  schedule  breach  is  slippage  of  six  months  or  more  from  the  target  baseline. 

11  Lack,  2012a.  An  APB  procurement  cost  breach  occurs  when  cumulative  program  cost  increases  greater  than 
or  equal  to  10  percent. 
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growth  in  RDT&E  and  procurement.  In  the  SARs,  there  are  occasional  downward 
cost  adjustments,  which  are  not  directly  connected  to  the  root  causes  but  are  captured 
in  the  analysis  process.  In  this  report,  we  summarize  the  net  effect  of  these  changes  in 
the  following  figures.  Figure  5.6  shows  that  there  was  almost  $50  million  in  downward 
adjustments  in  the  SARs,  which  includes  estimating  (adjustment  for  current  and  prior 
escalation;  budget  cuts  from  DoD  and  Congress).  An  additional  $71.6  million  was 
associated  with  underestimating  systems  engineering,  integration,  and  testing  (SEIT) 
in  the  Inc.  1A  program  and  therefore  contract  cost  growth.  This  $71.6  million  was 
directly  related  to  meeting  the  original  requirements  outlined  in  JPALS  Increment  1  A. 
RDT&E  grew  an  additional  $334.6  million  because  of  the  restructuring  of  the  pro¬ 
gram  leading  to  increased  scope  from  Increments  3  and  4.  Increments  3  and  4  were 
not  fully  defined,  so  the  incorporation  of  these  dollar  values  into  Increment  1A  intro¬ 
duces  new  uncertainty.  For  instance,  the  period  of  development  has  been  extended  in 
the  restructure  at  a  cost  of  $215  million.  New  requirements  associated  with  automatic 
landing  capabilities  increased  the  RDT&E  costs  approximately  $120  million  more. 

For  procurement,  the  cost  adjustments  in  the  SARs  reveal  a  similar  pattern  with 
a  few  costs  attributable  to  the  original  scope  of  JPALS  Inc.  1A  but  the  majority  of 

Figure  5.5 

Drilling  Down  from  Root  Causes  to  Cost  Drivers 


FAA  decisions  regarding 
precision  approaching 
and  landing 


Program 

interdependencies 


Restructure  to  include 
technical  requirements 
from  Incs.  3  &  4 

Increased  scope 

Restructure:  change  in 
shore  requirement 

Reduction  in  quantity 

Underestimation  of  SEIT 
(some  attributable  to 
FAA  and 

interdependencies) 

Contract  cost  growth 

Test  asset  availability 

Stretched  schedule 

SOURCE:  RAND  analysis. 

NOTES:  Blue  is  a  root  cause.  Gray  is  technical  factor  or  other  change  in  the  program  based  on  a  root 
cause.  Green  is  the  consequence. 


RAND  MG1 171/8-5.5 


32  Root  Cause  Analysis 


Figure  5.6 

Program  Restructure  Drove  RDT&E  Cost  Increase,  2009  Through  2013 
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SOURCES:  RAND  analysis;  Department  of  Defense,  JPALS  Inc.  1A  SARs,  December  2009  through  2013. 
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changes  associated  with  restructuring  the  program  as  exhibited  in  Figure  5.7.  Contract 
cost  growth  and  stretched  schedule  added  about  $19  million.  Procurement  decreased 
by  $70  million  because  of  the  reduction  in  shore  infrastructure  by  a  quantity  of  10. 
Increasing  the  scope  of  the  current  JPALS  program  of  record  added  an  additional  $165 
million  because  of  shifting  government  staff  from  later  increments  and  other  small 
restructuring  costs. 

Once  all  of  the  cost  variances  from  the  SARs  are  categorized,  it  is  possible  to  sum¬ 
marize  the  cost  driver  effect  at  the  unit  cost  level  and  associate  each  with  a  percentage. 
This  gives  us  a  better  understanding  of  the  increase  in  costs  based  on  spreading  those 
costs  over  fewer  units.  The  first  step  is  to  identify  the  baseline  cost  of  interest  from  the 
SAR,  either  the  PAUC  or  APUC.  Figure  5.8  includes  PAUC  and,  therefore,  for  this 
calculation  we  used  the  total  program  cost  and  the  cost  variances  for  RDT&E  and 
procurement  combined.  Then  we  took  the  total  cost  and  subtracted  the  cost  variance 
of  interest  (contract  cost  growth,  stretch  schedule,  increased  scope,  etc.).  Then  this 
number  is  divided  by  the  new  quantity.  Finally,  we  take  this  number  and  subtract  the 
baseline  PAUC  to  see  the  increase  in  costs  based  on  spreading  cost  increases  over  fewer 
units. 

These  calculations  reinforce  the  notion  that  stretched  schedule  and  contract  cost 
growth  are  minor  drivers  of  JPALS  cost  growth  and  that  the  reduction  in  quantity  and 
increased  scope  have  accounted  for  over  85  percent  of  cost  growth. 

The  next  unit  cost  driver  calculation  is  similar  to  the  prior  one  but  is  limited  to 
procurement  costs  because  its  baseline  is  APUC.  First,  we  identified  the  total  cost 
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Figure  5.7 

Increased  Scope  from  Incs.  3  and  4  Drove  Procurement  Cost  Increases,  2009  Through  2013 
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SOURCES:  RAND  analysis;  Department  of  Defense,  JPALS  Inc.  1 A  SARs,  December  2009  through  2013. 
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Figure  5.8 

Scope  Change  Is  the  Major  Driver,  But  Lower  Quantity  Means  That  RDT81E  Is  Spread  over 
Fewer  Units  in  the  PAUC  Metric 
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SOURCES:  RAND  analysis;  Department  of  Defense,  JPALS  Inc.  1A  SARs,  December  2009  through  2013. 
NOTE:  Numbers  may  not  add  to  100  percent  because  of  rounding. 
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associated  with  procurement  and  the  cost  variances  associated  with  procurement  dol¬ 
lars.  Then,  we  took  the  total  cost  and  subtracted  the  cost  variance  of  interest.  Next, 
we  divided  this  number  by  the  new  quantity,  and,  finally,  we  subtracted  the  baseline 
APUC.  The  resulting  number  is  the  effect  of  that  cost  variance  on  the  procurement 
unit  cost.  As  shown  in  Figure  5.9,  it  appears  from  the  data  that  the  shore-based  facili¬ 
ties  that  were  eliminated  during  the  restructuring  may  have  cost  more  per  unit  than 
the  ship-based  counterpart.  Therefore  the  cost  associated  with  reducing  the  quantity 
in  this  case  is  unusually  negative.  Of  course,  the  smaller  quantity  did  mean  that  other 
cost  variances  had  a  great  influence  on  unit  cost. 

For  the  APUC,  contract  cost  growth  and  stretched  schedule  have  a  minimal  effect 
on  unit  cost  growth.  Increased  scope  shifted  new  dollars  from  other  increments  and 
therefore  accounts  for  90  percent  of  the  unit  cost  growth. 


Limitations 

After  our  initial  analysis,  PARCA  used  these  findings  along  with  the  unpublished  cost 
estimates  developed  by  cost  analysts  at  the  office  of  Cost  Assessment  and  Program  Eval- 

Figure  5.9 

About  90  percent  of  APUC  Growth  Was  Driven  by  Increased  Scope 
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SOURCES:  RAND  analysis;  Department  of  Defense,  JPALS  Inc.  1A  SARs,  December  2009  through  2013. 
NOTE:  Numbers  may  not  add  to  100  percent  because  of  rounding. 
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uation  (CAPE)  to  develop  the  official  root  cause  analysis  memo.12  These  new  CAPE 
cost  estimates  gathered  new  information  from  the  program  and  the  contractors  beyond 
what  was  available  at  the  time  of  SAR  publication.  This  additional  estimate  allowed 
PARCA  to  refine  the  percentage  of  cost  growth  related  to  each  driver  with  information 
beyond  what  was  available  in  the  SAR.  Therefore,  the  final  cost  growth  percentages  in 
the  memo  deviate  slightly  from  our  characterization  here,  but  the  direction  and  mag¬ 
nitude  of  cost  growth  attributed  to  the  root  causes  are  similar  to  those  in  our  findings. 


12  Gary  R.  Bliss,  “Root  Cause  Analysis  of  the  Joint  Precision  Approach  and  Landing  System  increment  1A 
(JPALS  Inc  1A)  Program,”  memorandum  for  the  Under  Secretary  of  Defense  (AT&L),  Washington  D.C., 
May  23,  2014. 


CHAPTER  SIX 

Conclusions 


The  JPALS  program  is  a  joint  acquisition  effort  based  on  an  important  assumption  that 
the  FAA  and  DoD  would  collectively  transition  to  a  GPS-based  precision  approach 
and  landing  capability,  which  proved  to  be  false.  Between  2005  and  2012,  the  timing 
of  the  FAA  decision  to  transition  to  GPS  from  the  ILS  standard  for  PALC  became 
uncertain.  As  a  result,  the  Air  Force  and  Army  reassessed  JPALS  Inc.  2  because  of  the 
FAA  decision  to  maintain  ILS  as  standard  and  an  increasingly  tight  budget  environ¬ 
ment.  Army  funding  for  JPALS  ended  in  FY  2012,  and  Air  Force  funding  ended  in  FY 
2013  followed  by  a  SAF/AQ  memo  stating  that  the  Air  Force  would  not  move  forward 
with  JPALS  Inc.  2.  Taking  into  account  the  decisions  by  the  FAA,  the  Army,  and  the 
Air  Force,  the  Navy  revisited  its  PALC  plans.  The  PALC  assessment  concluded  that 
Inc.  IB  was  unaffordable  for  the  Navy  and  that,  over  time,  there  would  be  changes 
in  capability  gaps  in  existing  systems.  The  Navy’s  PALC  study  also  determined  that  a 
scope  change  for  JPALS  was  necessary  because  of  fact-of-life  changes  and  affordabil¬ 
ity.  The  Navy  proposed  bringing  Incs  3  and  4  into  1A  and  consolidating  funding  for 
Incs  1A,  IB,  and  2.  More  specifically,  the  reduction  in  total  planned  quantities  in  the 
Inc.  1A  program  because  of  the  elimination  of  previously  required  shore-based  train¬ 
ing  systems,  an  extension  of  the  development  program  to  include  capability  improve¬ 
ments,  a  lower  and  longer  procurement  profile,  and  increases  in  material  costs  resulted 
in  a  critical  Nunn-McCurdy  unit  cost  breach  to  the  PAUC  and  APUC  in  the  current 
JPALS  Inc.  1A  program. 

The  cost  drivers  for  the  critical  unit  cost  breaches  of  both  the  PAUC  and  APUC 
against  both  baselines  in  order  of  importance  are  as  follows: 

•  increased  program  scope/requirements  change 

•  engineering  and  manufacturing  development  (EMD)  extension  to  incorporate 
Incs.  3  and  4  into  Inc.  1A 

•  consolidated  funding  for  staff  and  extended  coverage  under  a  restructured  single 
increment  program 

•  quantity  reduction  spreading  nonrecurring  costs  over  fewer  units 

•  schedule  stretch  for  Inc.  1A  and  flatter  production  profile 
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•  underestimation  of  the  SEIT  effort  because  the  algorithms  were  more  difficult 

than  anticipated. 

The  JPALS  Inc.  1A  program  also  experienced  SEIT  asset  availability  challenges 
(particularly  with  the  CVN  testing)  that  resulted  in  cost  growth  of  approximately 
10  percent  including: 

•  movement  of  one  RDT&E  unit  to  procurement 

•  APB  schedule  and  procurement  cost  breaches 

•  some  unit  cost  growth. 

However,  the  10  percent  growth  in  the  JPALS  Inc.  1A  program  would  not  have 
caused  any  Nunn-McCurdy  breaches. 

After  recertification  of  a  program  with  a  critical  Nunn-McCurdy  breach,  PARCA 
is  responsible  for  continued  monitoring  of  the  program’s  performance  to  provide  situ¬ 
ational  awareness  of  future  execution  issues  and  guard  against  repeat  Nunn-McCurdy 
breaches.  During  the  course  of  our  root  cause  analysis,  we  identified  a  set  of  risks  with 
the  potential  to  cause  a  future  breach.  This  set  is  not  intended  to  be  a  comprehensive 
list  of  future  risks  for  the  restructured  JPALS  program,  and  we  do  not  attempt  to  assess 
the  relative  importance  or  consequences  of  these  risks.  Rather,  we  offer  this  discussion 
of  potential  future  risk  as  a  starting  point  for  PARCA’s  continued  monitoring  on  the 
program. 

The  development  risk  associated  with  the  incorporation  of  previously  defined 
Incs.  3  and  4  capabilities  into  the  restructured  JPALS  program  is  one  future  risk.  We 
did  not  examine  the  capabilities  and  technologies  involved  specifically,  but  developing 
increased  capabilities — capabilities  that  increase  or  add  to  performance  of  the  base¬ 
line  system — generally  presents  some  degree  of  technical  or  programmatic  risk.  The 
risk  may  manifest  as  poor  or  incomplete  requirements  definition  and  translation  into 
performance  specifications,  unanticipated  software  development  or  system  integration 
challenges,  or  even  future  budget  reductions  or  reduction  in  demand  for  the  future 
capability. 

A  related  risk  is  that,  at  the  time  of  this  writing,  the  technical  baseline  for  what 
was  Inc.  4  capability  had  not  been  sufficiently  defined.  This  capability  is  intended 
to  enable  precision  landing  of  UAVs  and  is  part  of  the  restructured  JPALS  program. 
A  poorly  or  incompletely  defined  technical  baseline  introduces  significant  risk  into 
the  cost  and  schedule  estimates  that  will  be  baselined  at  Milestone  B  after  program 
recertification. 

The  requirement  for  the  capabilities  that  the  restructured  JPALS  program  pro¬ 
duces  appears  to  be  limited  to  the  L-35B/C  and  UCLASS  (or  other  future  sea-based 
UAVs).  Slips  in  the  L-35  B/C  program  could  adversely  affect  JPALS  cost  and  schedule. 
Similarly,  the  UCLASS  program  has  not  been  fully  defined,  and  its  demand  for  JPALS 
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has  not  been  formalized  into  an  approved  requirement.  Changes  in  the  deployment 
timing  or  quantities  could  lead  to  changes  in  the  JPALS  program.  This  is  really  just  a 
continuation  of  the  program  interdependencies  factor  discussed  above. 

As  noted  above,  performance  of  the  JPALS  Inc.  1A  capability  has  largely  been 
demonstrated  using  JPALS  EDMs  and  representative  aircraft  integration  kits  installed 
on  C-12  or  F/A-18  aircraft.  Recent  testing  has  also  begun  to  demonstrate  some  of  the 
Inc.  3  capabilities  that  will  become  part  of  the  restructured  program.  Because  the  oper¬ 
ational  need  for  JPALS  is  linked  to  the  deployment  of  the  F-35B/C  and  UCLASS,  and 
those  aircraft  are  not  expected  to  become  operational  for  several  years,  the  restructured 
JPALS  program  includes  a  production  gap  based  on  the  restructured  JPALS  program’s 
new  Milestone  C  and  low-rate  initial  production  (LRIP)  dates.  This  suggests  a  possible 
loss  of  learning  affecting  manufacturing  and  integration  of  the  JPALS  system,  as  well 
as  the  risk  of  hardware  obsolescence.  Either  of  these  risks  associated  with  the  extended 
scheduled  of  the  restructured  JPALS  program  could  affect  program  cost  in  the  future. 
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A  Methodology  for  Assessing  the  Department  of  Defense 
Acquisition  Portfolio 


Developing  an  Approach  to  Portfolio  Assessment 

For  the  purposes  of  oversight  and  monitoring,  planning  and  programming,  and  deci¬ 
sionmaking  regarding  development  and  procurement,  policymakers  and  leaders  of  the 
defense  acquisition  community  need  to  be  able  to  describe  and  assess  the  status  and 
performance  of  portfolios  of  MDAPs  over  time.  For  example,  they  may  need  to  evalu¬ 
ate  the  maturity,  average  unit  cost  growth,  amount  of  churn  (or  number  of  programs 
with  a  quantity  change),  or  trends  in  funds  remaining  of  a  specific  category  or  group  of 
programs  (e.g.,  helicopters,  satellite,  ships,  fighter  jets,  and  others)  and  be  able  to  sum¬ 
marize  the  performance  of  these  programs  using  a  clear  and  easy  to  understand  set  of 
metrics  and  visualizations.  They  may  also  be  interested  in  how  this  portfolio’s  perfor¬ 
mance  changes  over  time,  how  its  performance  compares  to  other  portfolios,  and  even 
how  the  composition  of  the  portfolio  itself  may  be  evolving  (e.g.,  which  programs  have 
been  completed  or  cancelled,  and  which  are  still  ongoing). 

This  chapter  presents  a  methodology  and  set  of  metrics  that  can  be  used  to  char¬ 
acterize  the  status  and  risk  of  portfolios  and  subportfolios  of  MDAPs  over  time  and 
across  commodity  types.  It  responds  to  an  interest  expressed  by  PARCA  in  developing 
a  clear  and  repeatable  way  to  assess  and  summarize  the  overall  performance  of  a  set  of 
acquisition  programs  at  a  given  point  in  time  and  over  a  longer  time  span.  The  primary 
audience  for  this  part  of  the  report  consists  of  analysts  in  the  acquisition  community 
who  might  use  the  metrics  and  visualizations  to  understand  the  performance  of  many 
portfolios  across  several  different  dimensions.  The  assessments  and  insights  derived 
from  these  visualizations  may  also  be  of  use  to  policymakers.  However,  it  is  worth 
noting  that  the  methodology  itself  is  the  most  significant  contribution  of  our  discus¬ 
sion  of  portfolio  assessment  and  metrics. 

To  begin,  it  is  necessary  to  define  what  is  meant  by  the  term  portfolio  in  the  con¬ 
text  of  this  chapter.  We  use  the  term  portfolio  to  refer  to  a  set  of  acquisition  programs 
that  can  be  grouped  together  because  they  share  a  certain  characteristic  or  several 
characteristics.  For  example,  the  helicopter  portfolio  is  a  set  of  all  helicopter  programs. 
Portfolios  can  be  designed  and  constructed  in  any  number  of  ways  depending  on  the 
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interests  of  analysts.  For  instance,  portfolios  might  be  based  on  Defense  Acquisition 
Management  Information  Retrieval  (DAMIR)  program  types  (all  force  application 
or  logistics  programs,  for  example).  Or,  they  might  be  based  on  the  agency  or  service 
that  owns  the  program.  These  portfolios  would  include  all  Army,  all  Navy,  or  all  DoD 
programs.  Another  type  of  portfolio  might  be  constructed  around  individual  contrac¬ 
tors,  with  each  portfolio  including  all  the  programs  currently  managed  by  key  DoD 
contractors.  Finally,  a  portfolio  might  be  organized  to  include  all  programs  at  certain 
points  in  the  program  life  cycle,  past  Milestone  C  or  approaching  Milestone  B,  for 
example.  We  refer  also  in  the  report  to  subportfolios.  Subportfolios  include  smaller  sets 
of  MDAPs  with  similar  characteristics.  The  analysis  of  a  subportfolio  allows  us  to  study 
the  status  of  these  smaller  sets  of  programs  in  more  detail  and  to  separate  their  status 
from  the  status  of  the  overall  portfolio  of  MDAPs  and  the  status  of  other  subportfolios. 

There  is  no  set  methodology  for  constructing  a  portfolio  of  programs.  Instead,  the 
construction  will  be  based  on  the  interests  and  priorities  of  policymakers  and  analysts. 
For  example,  analysts  interested  in  understanding  the  overall  performance  of  all  pro¬ 
grams  owned  by  each  service  will  want  to  construct  larger  portfolios  based  around  each 
individual  service.  In  contrast,  an  analyst  interested  in  understanding  the  performance 
of  individual  program  types  within  the  Army’s  larger  portfolio  of  programs  might  con¬ 
struct  and  analyze  subportfolios  of  programs  including  aircraft,  wheeled  vehicles,  and 
weapons.  Portfolios  may  even  be  formed  around  sets  of  programs  that  are  more  differ¬ 
ent  than  alike.  For  example,  “all  programs  experiencing  a  cost  breach  in  past  five  years” 
would  be  a  possible  portfolio  of  programs,  and  the  resulting  portfolio  would  contain 
significant  diversity.  However,  ultimately,  the  analysis  of  this  portfolio  would  be  useful 
in  assessing  the  overall  performance  and  continued  risk  of  this  set  of  programs.  Thus, 
the  construction  of  a  portfolio  is  a  flexible  concept  that  can  be  molded  to  address  the 
specific  questions  asked  by  an  analyst  or  policymaker.  This  flexibility  increases  the  util¬ 
ity  of  portfolio-level  analysis  for  the  acquisition  community. 

There  are  no  size  requirements  when  defining  and  constructing  portfolios  of  pro¬ 
grams.  Instead,  what  is  most  important  is  that  all  programs  in  a  given  portfolio  share 
key  characteristics  and  are  included  in  the  portfolio  for  specific  reasons.  A  portfolio 
of  programs  could  contain  hundreds  of  programs  or  only  a  few.  Different  methodolo¬ 
gies  for  assessing  portfolio  performance  might  be  used  for  different  size  portfolios.  The 
methodology  described  in  this  report  will  likely  be  most  useful  when  the  number  of 
programs  is  relatively  small — probably  less  than  25  programs  (but  more  than  five  or 
so).  This  is  because  the  set  of  metrics  included  are  relatively  extensive,  and  some  of  the 
visualizations  are  complicated  to  create  when  the  number  of  programs  becomes  too 
large.  Also,  when  portfolios  are  too  large,  it  may  be  difficult  to  drill  down  and  explain 
the  drivers  of  key  portfolio-level  trends.  As  will  become  clear  in  our  sample  portfolios, 
whereas  the  first  step  of  portfolio  analysis  requires  looking  at  summary  trends,  the 
second  step  involves  digging  deeper  and  understanding  the  drivers  of  those  trends.  This 
is  harder  to  do  when  there  are  so  many  different  programs.  However,  when  there  are 
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too  few  programs,  the  portfolio  view  is  also  less  useful,  because  it  will  be  similar  to  a 
program-level  analysis  of  any  individual  program  and  is  likely  to  be  easily  influenced 
by  individual  programs  within  the  portfolio. 

There  are  several  advantages  to  studying  acquisition  program  performance  at 
the  portfolio  level.  A  summary  analysis  can  provide  a  broader  insight  into  the  per¬ 
formance  of  a  set  of  acquisition  programs — current  status,  future  risks,  and  overall 
performance  trend.  As  a  result,  it  gives  a  more  accurate  picture  of  overall  acquisi¬ 
tion  performance  than  focusing  on  individual  programs,  be  they  strong  performers 
or  programs  with  significant  problems  and  challenges.  The  portfolio-level  analysis  can 
also  identify  programs  and  portfolios  that  are  at  risk,  that  need  additional  investment, 
or  that  are  performing  particularly  well.  Although  it  is  difficult  to  compare  perfor¬ 
mance  across  some  portfolios  given  their  extensive  differences  (comparing  helicopters 
to  wheeled  vehicles,  for  example),  comparisons  across  other  types  of  portfolios  might 
be  more  meaningful  (for  example,  comparing  Army  programs  to  Navy  programs). 
This  type  of  comparison  might  help  acquisition  analysts  identify  broader  families  or 
types  of  programs  that  are  more  or  less  at  risk  for  performance  problems  or  that  have 
contributed  more  or  less  to  such  problems  as  cost  growth  and  slippage  in  the  past.  A 
portfolio-level  assessment  can  also  help  acquisition  analysts  understand  the  ways  in 
which  a  given  portfolio  may  be  at  risk.  For  example,  a  portfolio  that  has  high  and 
increasing  cost  growth  may  pose  a  different  kind  of  risk  than  one  with  low  cost  growth 
but  faster  than  expected  RDT&E  or  procurement  spending.  Being  able  to  diagnose 
which  portfolios  are  performing  well  and  which  pose  more  risk  (and  what  type  of  risk 
they  pose)  can  help  acquisition  analysts  manage  the  overall  risk  incurred  by  DoD  and 
plan  for  future  acquisitions. 

For  the  reasons  noted  above,  analysis  of  acquisition  performance  conducted  at 
the  portfolio  level  may  be  useful.  To  facilitate  this  type  of  analysis,  we  have  focused 
in  this  report  on  the  development  of  a  methodology  and  a  set  of  metrics  that  can  be 
used  to  conduct  portfolio  analysis.  Our  focus  throughout  the  report  is  on  defining 
and  discussing  metrics  and  visualizations  of  these  metrics  that  can  be  used  as  part 
of  a  valuable  portfolio  assessment.  We  discuss  the  types  of  data  sources  that  can  be 
used,  challenges  associated  with  these  data  sources,  and  ways  to  overcome  these  chal¬ 
lenges.  We  also  provide  clear  rationale  for  why  we  have  chosen  the  metrics  that  we  have 
included  and  suggest  a  range  of  different  metrics  and  visualizations  that  may  be  useful 
to  analysts  with  varied  interests  and  objectives.  We  then  use  two  example  portfolios  to 
demonstrate  and  refine  our  chosen  methodology:  the  satellite  and  helicopter  portfolios, 
from  2002  to  2012.  Our  example  portfolios  allow  us  to  validate  and  demonstrate  our 
methodology  and  to  provide  and  interpret  data  visualizations  that  a  policymaker  could 
use  to  understand  and  describe  the  status  of  various  portfolios  and  subportfolios.  We 
use  one  of  these  portfolios,  the  helicopter  portfolio,  to  conduct  a  more  in-depth  assess¬ 
ment,  demonstrating  how  our  methodology  and  metrics  can  be  applied  and  inter- 
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preted  to  provide  an  expanded  assessment  of  portfolio  status  and  performance.1  How¬ 
ever,  throughout  the  chapter  our  focus  remains  the  definition  of  a  methodology  and 
selection  and  discussion  of  the  metrics  and  visualizations  included  in  this  methodol¬ 
ogy.  We  do  not  provide  a  comprehensive  portfolio  assessment  but  rather  offer  analysts 
an  approach  and  a  set  of  tools  and  visualizations  that  they  can  use  to  do  so. 

It  is  worth  noting  that  other  methodologies  could  be  and  already  are  used  to  assess 
program  and  portfolio  performance.  For  example,  the  “Probability  of  Success”  Method 
used  by  each  service  is  one  approach  to  evaluating  the  performance  and  risk  inherent 
in  a  given  portfolio  of  programs.2  That  model  evaluates  programs  on  five  dimensions 
requirements,  resources,  program  execution,  fit  in  the  vision,  and  advocacy.  The  met¬ 
rics  can  also  be  used  to  assess  the  “probability  of  success”  for  individual  programs.  As 
with  the  portfolio  assessment  methodology  presented  in  this  report,  the  probability  of 
success  model  considers  many  aspects  of  program  performance  (beyond  cost,  schedule, 
and  performance)  and  uses  graphical  presentations  to  summarize  performance  trends. 
However,  the  focus  of  the  methodology  presented  in  this  report  is  not  to  assess  the  per¬ 
formance  of  individual  programs  but  to  provide  aggregate,  portfolio-level  views  that 
provide  a  summary  of  the  status  of  an  entire  portfolio  and  trends  over  time. 


Organization  of  Part  Two 

Chapter  Eight  documents  our  methodology  for  constructing  a  portfolio  analysis, 
including  defining  objectives,  selecting  a  portfolio,  vetting  data  and  metrics,  and  visu¬ 
alizing  the  results.  To  illustrate  this  methodology,  we  include  examples  from  two  differ¬ 
ent  portfolios  we  examined  in  more  depth:  the  helicopter  and  satellite  programs  from 
2002  to  2012.  Most  important,  the  methodological  approach  we  used  built  on  existing 
related  work  on  anticipating  breaches  that  had  already  reviewed  a  large  number  of  met¬ 
rics  and  performed  data  collection  and  processing  on  a  set  of  acquisition  programs.3 
Chapter  Nine  presents  a  final  outcome  for  our  own  constructed  portfolio  analysis  of 
helicopter  and  satellite  programs,  which  includes  the  set  of  visualizations  that  indicate 
status,  trends,  and  patterns  with  the  portfolio  metrics.  We  focus  in  this  chapter  on 
our  methodology,  metrics,  and  visualizations  rather  than  giving  a  complete  assessment 
of  the  programs  in  the  portfolios  that  we  consider.  Chapter  Ten  provides  an  example 


1  In  Appendix  B,  we  provide  information  on  an  initial  “test  case”  portfolio  that  we  used  in  developing  the 
methodology. 

2  For  more  on  the  Probability  of  Success  Method,  see  John  Higbee  and  LTC  Robert  Ordonio,  “Program  Suc¬ 
cess:  A  Different  Way  to  Assess  It,”  AT&L  Magazine,  May  2005. 

3  Mark  V.  Arena,  John  Birkler,  Irv  Blickstein,  Charles  Nemfakos,  Abby  Doll,  Jeffrey  A.  Drezner,  Gordon  T. 
Lee,  Megan  McKernan,  Brian  Mclnnis,  Carter  C.  Price,  Jerry  M.  Sollinger,  and  Erin  York,  Management  Perspec¬ 
tives  Pertaining  to  Root  Cause  Analyses  of  Nunn-McCurdy  Breaches,  Volume  6:  Contractor  Motivations  and  Antici¬ 
pating  Breaches,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-1171/6-OSD,  2014. 
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followup  narrative  that  may  result  from  additional  questions  from  a  policymaker 
inspired  by  the  results  of  the  initial  portfolio  assessment.  In  the  concluding  chapter 
(Chapter  Eleven),  we  describe  the  limitations  and  next  steps  in  the  research,  including 
performing  a  more  complete  portfolio  assessment. 


CHAPTER  EIGHT 


Construction  of  a  Portfolio  Analysis:  Objectives,  Portfolios, 
Data,  Metrics,  and  Visualization  Selection 


In  this  chapter,  we  present  a  general  framework  to  construct  a  portfolio  analysis  that 
provides  flexibility  to  incorporate  policymakers’  priorities  and  assumptions.  As  a  means 
of  demonstration,  we  provide  examples  of  three  portfolio  frameworks  we  examined  as 
we  worked  to  fine-tune  and  develop  this  method.  After  identifying  analysis  objectives 
and  assumptions,  we  discuss  the  selection  of  portfolios  and  data  sources,  providing 
information  about  the  selection  of  metrics  at  the  program  and  portfolio  levels.  We  also 
discuss  possibilities  for  data  packaging  and  visualization  techniques  that  we  used  in 
our  analysis.  The  following  chapters  then  expand  on  our  assessment  of  our  example 
portfolios.  Because  the  goal  of  the  initial  phase  of  this  work  was  to  develop  a  method¬ 
ology  that  can  be  used  to  assess  portfolio  status,  we  spent  a  considerable  effort  on  this 
first  step  of  the  framework  construction  to  document  our  rationale  for  each  method¬ 
ological  step  and  decision  in  this  chapter  of  the  report. 


Identifying  Portfolio  Analysis  Objectives 

Before  beginning  a  portfolio  analysis,  it  is  important  to  outline  the  objectives  of  the 
analyst  or,  more  important,  of  the  policymaker  in  taking  a  portfolio  view  as  opposed 
to  an  individual  program  view.  These  objectives  may  range  from  the  priorities  of  an 
individual  policymaker  to  tracking  the  success  of  DoD-wide  acquisition  policies.  For 
example,  by  taking  the  portfolio  view,  one  may  assess  the  portfolio-wide  implementa¬ 
tion  and  results  of  implemented  policy,  such  as  Better  Buying  Power  2.0. 1  From  these 
identified  objectives  flow  a  set  of  assumptions  as  to  how  one  rates  the  “performance”  of 
a  chosen  portfolio.  Again,  to  build  from  the  Better  Buying  Power  2.0  example,  acqui¬ 
sition  officials  have  adopted  “affordability  constraints”  to  track  individual  programs, 


1  Initiated  in  2012  by  Frank  Kendall,  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics. 
The  initial  version  of  Better  Buying  Power  3.0  was  released  in  September  2014.  See  Frank  Kendall,  Under  Secre¬ 
tary  of  Defense  Acquisition,  Technology  and  Logistics,  “Better  Buying  Power  3.0:  White  Paper,”  September  19, 


2014. 
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such  as  affordability  unit  cost  and  operations  and  sustainment  (O&S)  goals  and  cap 
metrics,  which  allow  for  the  grading  of  program  performance  in  these  areas.2  These 
predefined  metrics  (for  example,  if  unit  cost  rises  above  a  certain  value,  this  signals  bad 
performance)  help  perform  quick  assessments  on  programs  and,  in  the  aggregate,  the 
acquisition  portfolio.  Although  outlining  metric  assumptions  derived  from  the  port¬ 
folio  analysis’s  objectives  help  to  scope  the  analysis,  one  must  always  use  the  objec¬ 
tive  itself  as  a  frame  of  reference,  as  patterns  or  trends  within  the  data  may  point  to 
negative  or  positive  implications  not  covered  by  predefined  assumptions.  Within  our 
own  example  portfolios,  our  main  objectives  were  to  identify  cost  and  schedule  risk 
indicators. 

Objectives  for  the  portfolio  analysis  will  be  heavily  influenced  by  the  choice  of 
the  portfolio  itself.  For  the  purposes  of  this  report,  we  define  a  portfolio  as  a  group  of 
acquisition  programs  sharing  a  defined  set  of  characteristics.  The  possibilities  for  port¬ 
folio  selection  within  defense  acquisitions  are  numerous.  Several  examples  for  defined 
portfolio  “types”  include  acquisition  type  (MDAP,  Major  Acquisition  Information 
System  [MAIS],  etc.),  service,  commodity  type,  size  (in  dollars),  contractor,  contract 
type,  time  frame,  or  other  characteristics  that  may  be  of  interest  to  a  policymaker.  If 
they  share  the  same  “type,”  multiple,  analogous  portfolios  may  then  be  compared,  such 
as  the  Air  Force  acquisition  portfolio  versus  the  Navy  acquisition  portfolio. 

After  developing  and  refining  our  methodology  using  a  test  case  portfolio  of 
MDAP  programs  (see  Appendix  B),  we  applied  the  methodology  to  two  commodity- 
specific  portfolios  over  a  period  of  11  years,  starting  in  2002  and  ending  in  2012.  The 
component  types  were  based  on  DAMIR  pre-dehned  “Type”  portfolios,  and  the  two 
we  selected  were  the  helicopter  programs  and  satellite  programs.  These  portfolios  con¬ 
tained  all  active  MDAPs  reporting  SAR  data  at  any  time  between  December  1,  2002, 
and  December  31,  2012. 

Part  of  our  motivation  for  selecting  the  satellite  portfolio  is  recent  interest  by 
the  sponsor  and  the  acquisition  community  in  evaluating  risk  in  satellite  programs.3 
The  satellite  portfolio  included  a  total  of  12  programs  between  2002  and  2012.  At 
any  given  time  during  that  period,  between  seven  and  nine  programs  were  active  and 
reporting  SAR  data.  A  majority  of  the  programs  in  this  portfolio  were  Air  Force  pro- 


2  Obtained  from  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics, 
“Affordability  Constraints  of  ACAT-ID  IAM  Programs  for  Components  9-15-14. xlsx,”  DAMIR  Files,  undated. 

3  Risks  and  cost  growth  of  satellite  programs  have  been  addressed  by  a  number  of  recent  publications,  including 
Myron  Hura,  Space  Capabilities  Development:  Continued  Difficulties  and  Suggested  Actions,  Santa  Monica,  Calif.: 
RAND  Corporation,  2011,  Not  available  to  the  general  public;  Obaid  Younossi,  Mark  A.  Lorell,  Kevin  Brancato, 
Cynthia  R.  Cook,  Mel  Eisman,  Bernard  Fox,  John  C.  Graser,  Yool  Kim,  Robert  S.  Leonard,  Shari  Lawrence 
Pfleeger,  and  Jerry  M.  Sollinger,  Improving  the  Cost  Estimation  of  Space  Systems:  Past  Lessons  and  Future  Recom¬ 
mendations ,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-690-AF,  2008;  Government  Accountability  Office, 
“Space  Acquisitions:  DOD  Faces  Challenges  in  Fully  Realizing  Benefits  of  Satellite  Acquisition  Improvements,” 
Washington,  D.C.,  GAO-12-563T,  2012;  Ellen  Pawlikowski,  “Space  Acquisition  Issues  in  2013,”  Air  &  Space 
Power  Journal,  September-October  2013. 
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grams.  Table  8.1  shows  all  the  programs  that  made  up  the  satellite  portfolio  by  year, 
service,  and  latest  milestone  achieved  by  year. 

Because  the  satellite  portfolio  includes  mainly  Air  Force  programs,  one  reason 
for  choosing  the  helicopter  portfolio  was  to  also  include  programs  under  the  Army’s 
control  and  a  more  representative  sample  of  programs  under  the  Navy’s  control.  The 
helicopter  portfolio  included  13  programs  between  2002  and  2012  with  between  seven 
and  10  programs  active  and  reporting  SAR  data  at  any  given  time.  A  full  list  of  the 
programs  in  the  helicopter  portfolio  by  year,  service,  and  by  the  last  milestone  achieved 
by  year  is  outlined  in  Table  8.2. 


Selecting  Data  and  Metrics 

The  identified  objectives  for  the  portfolio  analysis  should  frame  the  selection  of  data 
metrics  to  perform  the  assessment.  An  analyst  must  also  consider  the  ease  of  access  to 
the  data  as  well  as  the  amount  of  labor  required  to  gather  and  clean  the  data  for  the 


Table  8.1 

Satellite  Portfolio  Composition  Between  2002  and  2012 


Program 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

2010 

2011  2012 

Air  Force 

AEHF 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

GBS 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

GPS  III 

B 

B 

B 

C 

C 

GPS  OCX 

B 

NAVSTAR  GPS 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

NPOESS 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

SBIRS-High 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

SBSS 

B 

B 

B 

B 

TSAT 

B 

WGS 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

Navy 

MUOS 

B 

B 

C 

C 

C 

C 

C 

C 

C 

NESP 

C 

C 

C 

NOTES:  A  red  B  indicates  that  the  program  has  passed  Milestone  B,  and  a 

blue  C  indicates  that  the 

program  has  passed  Milestone  C.  A  blank  cell  in  the  table  suggests  that  the  program  has  not  started 
reporting  SAR  data,  has  been  cancelled,  or  is  no  longer  required  to  report  SAR  data. 
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Table  8.2 

Helicopter  Portfolio  Composition  Between  2002  and  2012 


Program 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

2010 

2011 

2012 

Army 

ARH 

B 

B 

B 

B 

CH-47F 

B 

B 

C 

C 

C 

C 

C 

C 

C 

C 

C 

Comanche 

B 

B 

UH-60M 

B 

B 

B 

C 

C 

C 

C 

c 

C 

c 

C 

AH-64D  Longbow  Apache3 

C 

C 

C 

C 

C 

C 

c 

c 

C 

LUH 

C 

C 

c 

c 

C 

c 

C 

AH-64E  Remanufacture3 

B 

B 

B 

B 

C 

c 

C 

AH-64E  New  Build3 

C 

c 

C 

CH-53K 

B 

B 

B 

B 

B 

B 

B 

B 

Navy 

H-1  Upgrades 

B 

B 

B 

B 

B 

B 

C 

C 

C 

C 

C 

MH-60R 

B 

B 

B 

B 

C 

C 

C 

C 

C 

C 

C 

MH-60S 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

c 

VH-71 

B 

B 

B 

B 

NOTES:  A  red  B  indicates  that  the  program  has  passed  Milestone  B,  and  a  blue  C  indicates  that  the 
program  has  passed  Milestone  C.  A  blank  cell  in  the  table  suggests  that  the  program  has  not  started 
reporting  SAR  data,  has  been  cancelled,  or  is  no  longer  required  to  report  SAR  data.  Our  team  chose 
these  three  portfolios  both  to  test  our  methodology  and  present  examples  for  future  analysts  to  follow 
in  their  own  portfolio  analyses.  Again,  one  may  choose  from  a  variety  of  acquisition  characteristics  to 
create  a  portfolio  or  a  set  of  analogous  portfolios  for  comparison. 

3  Three  separate  Apache  acquisition  programs  are  covered  in  this  analysis.  The  AH-64D  Longbow 
Apache  was  reported  in  the  SARs  from  1997  through  2010.  It  is  referred  to  as  "Longbow  Apache"  in 
subsequent  figures  in  this  analysis.  The  AH-64E  Remanufacture  started  its  SAR  reporting  in  September 
2006  under  the  name  of  "Longbow  Apache-Block  III  (AB3)."  It  then  was  reported  under  the  following 
names:  Apache  Block  III  (AB3),  Apache  Block  IIIA  (AB3A  Remanufacture),  and  finally  as  AH-64E  Apache 
Remanufacture  (AH-64E  Remanufacture)  in  December  2012.  Even  with  the  multiple  name  changes,  it 
remained  the  same  acquisition  program.  The  AH-64E  New  Build  started  in  2010  and  was  formed  based 
on  the  2009  Nunn-McCurdy  outcome  of  the  AH-64E  Remanufacture  program.  The  first  name  for  the 
New  Build  program  was  Apache  Block  NIB  New  Build  (AB3B  New  Build)  in  2010  and  was  changed  to 
AH-64E  Apache  New  Build  (AH-64E  New  Build)  in  2012. 

final  analysis.  Our  team  addressed  both  of  these  perspectives  in  parallel  as  we  deter¬ 
mined  our  own  portfolio  analysis  data  and  metrics.  First,  we  will  discuss  those  factors 
concerning  ease  of  access,  labor  required,  and  applicability  across  our  chosen  portfolio. 
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Data  Selection 

We  began  by  considering  the  characteristics  of  various  data  repositories  and  source 
documents  to  determine  which  would  best  support  a  repeatable,  lower  labor  portfo¬ 
lio  analysis  methodology.  Table  8.3  summaries  our  assessments  of  the  various  data 
sources. 

Bearing  in  mind  future  users  of  our  portfolio  analysis  methodology,  we  assessed 
data  repositories  by  accessibility,  applicability,  timeliness,  completeness,  and  required 
labor  for  data  collection.  The  data  repositories  considered  but  not  included  were  those 
listed  below: 

•  Service-Level  Acquisition  Databases:  These  include  the  Army  Acquisition  Infor¬ 
mation  Management  (AIM);  the  Navy’s  Research,  Development,  and  Acquisition 
Information  System  (RDAIS);  and  the  Air  Force’s  System  Metrics  and  Reporting 
Tool  (SMART).  During  our  initial  review  of  data  sources,  the  team  had  access 
only  to  SMART,  and  it  did  not  prove  to  be  sufficient  for  our  needs.  However,  it  is 
possible  that  AIM  and  RDAIS  contain  data  that  would  be  useful  in  the  comple¬ 
tion  of  portfolio  analysis  projects. 


Table  8.3 

Information  Systems  and  Selection  Criteria 


Service-Level 

Database  Selection  Acquisition 

Criteria  Databases  DCARC  AIR  DAMIR 


Applicability  of  available 
data  for  portfolio  metrics 


Ease  of  data  collection 


Timeliness/completeness 
of  data 


Accessibility  for  RAND 
project  team 


Accessibility  for  OSD  CAPE 
users 


NOTES:  Criteria  used  in  the  table  above  are  RAND-defined.  Each  cell  identifies  whether  a 
specific  data  source  met  that  specific  RAND  criterion.  Red  indicates  that  the  criteria  were 
not  fulfilled;  yellow  indicates  that  they  were  partially  filled;  green  indicates  that  they 
were  filled. 
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•  Defense  Cost  and  Resource  Center  (DCARC):  The  DCARC  portal  includes 
an  Earned  Value  Management  (EVM)  Repository  and  Contractor  Cost  Data 
Reports  (CCDRs)  in  Defense  Automated  Cost  Information  Management  System 
(DACIMS).  Although  DACIMS  proved  accessible  to  our  target  user  set,  the  data 
within  the  system  was  far  too  granular  for  our  metric  needs.  In  addition,  the 
downloading  of  large  amounts  of  data  on  many  MDAPs  proved  labor  intensive. 

•  Acquisition  Information  Repository  (AIR):  During  the  study,  this  database 
was  relatively  new  with  limited  access  and  incomplete  documentation  across  the 
MDAPs  within  the  portfolio.  As  the  information  system  becomes  more  popu¬ 
lated  with  documentation  over  time,  it  could  serve  as  a  useful  source  for  under¬ 
standing  many  additional  details  on  the  various  MDAPs  in  the  portfolio. 

Because  of  its  ease  of  access,  consistent  and  complete  documentation,  and  low 
labor  cost  in  retrieving  data,  the  team  concluded  that  DAMIR  served  as  the  most 
appropriate  database  for  the  portfolio  analysis.  Table  8.4  displays  the  acquisition  docu¬ 
ments  and  DAMIR-generated  reports  available  within  the  information  system. 

The  team  also  assessed  various  data  source  documentation  for  similar  criteria, 
including  ease  of  access  for  the  users,  consistency  and  validation  of  data,  and  the 
regularity  of  data  collection  and  presentation.  Taking  these  into  account,  the  team 

Table  8.4 

Documents  Available  in  DAMIR 


Acquisition  Documents 

DAMIR  Data  Captures 

APB 

SAR 

SAR 

SAR  baseline 

SAR  baseline 

DAES  quarterly  reports 

DAES 

DAES  reliability  growth  curve 

MAIS  annual  report  (MAR) 

DAES  risk  summary  and  issue  summary 

MAR  original  estimate 

Historical  critical  change  reports 

Program  Objective  Memorandum  (POM) 

MAR 

Budget  Estimation  Submission 

MAIS  quarterly  report 

Earned  Value  Analysis  (EVA) 

Sustainment  metrics 

Better  buying  power  goals  and  metrics 
Significant  acquisition  watchlist 

NOTES:  We  define  as  acquisition  documents  any  published  document  that  has 
information  on  a  program's  development  or  performance.  These  documents  are  not 
necessarily  available  all  years  for  all  programs.  The  official  acquisition  documents 
have  more  consistent  coverage  across  programs  than  do  the  DAMIR-generated 
documents.  The  data  sources  included  in  the  table  are  of  varying  quality.  For 
example,  the  data  in  SARs  are  generally  considered  of  better  quality  than  the  data 
in  the  DAES  reports.  Finally,  data  sources  that  are  both  acquisition  documents  and 
captured  in  DAMIR  are  included  in  both  columns. 
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concluded  that  SARs  would  serve  as  the  primary  source  documents  for  data  inputs. 
Congressionally  mandated,  the  SARs  are  heavily  vetted  and  produced  annually,  with 
occasional  quarterly  reports.1 * * 4  When  SARs  were  not  available  for  active  MDAP  pro¬ 
grams,  the  team  used  DAES  reports  from  December  of  the  corresponding  year  to  fill 
in  gaps  within  the  data.  Submitted  quarterly  to  the  service-level  acquisition  databases 
and  pulled  into  DAMIR,  DAES  include  not  only  program  status  and  assessment  infor¬ 
mation  by  the  program  managers  but  also  independent  assessments  by  OSD  and  Joint 
staff  stakeholders.5  The  team  used  DAMIR,  SARs,  and  DAES  for  all  of  the  portfolio 
cases,  including  the  test  case,  helicopter,  and  satellite  portfolios. 

Metric  Selection 

Our  portfolio  analysis  objectives — to  identify  and  characterize  cost  and  schedule 
risks — guided  our  selection  of  program  and  portfolio  metrics.  First,  we  wanted  to 
comprehensively  capture  the  different  types  of  risk  that  might  affect  acquisition  pro¬ 
gram  cost  and  schedule  outcomes  because  program  outcomes  may  be  affected  by  any 
number  of  types  of  risk.  Second,  we  thought  about  program  and  portfolio  indica¬ 
tors  together,  intending  to  select  program  data  that  we  could  then  aggregate  to  the 
portfolio-level  metrics.  Finally,  we  wanted  to  identify  and  use  existing  data  that  were 
relatively  easy  to  obtain  consistently  for  all  programs  within  the  portfolio.  It  is  worth 
noting  again  that  our  focus  here  was  at  the  portfolio  level.  We  wanted  to  select  metrics 
that  could  be  aggregated  into  portfolio-level  views  to  provide  insight  into  the  perfor¬ 
mance  of  the  portfolio  overall.  Portfolio-level  metrics  might  not  directly  identify  pro¬ 
grams  that  need  attention  but  can  be  used  to  view  portfolio  performance  trends  over 
time  and  to  compare  across  portfolios. 

The  first  step  in  our  selection  of  program-  and  portfolio-level  metrics  was  to  define 
the  types  of  risk  and  the  specific  outcomes  that  we  wanted  to  include  in  our  assess¬ 
ment.  We  focused  on  dimensions  along  which  programs  may  experience  problems, 
challenges,  or  setbacks  that  could  affect  program-level  cost  and  schedule  outcomes. 
These  risk  dimensions  are  listed  in  Table  8.5.  We  also  identified  descriptive  indicators 
at  the  program  level  that  would  indicate  outcomes  of  interest  to  the  decisionmaker.  It 
is  important  to  note  that  some  indicators,  such  as  schedule  change,  quantity  change, 
and  cost  growth,  can  result  from  multiple  risk  factors.  There  is  no  one-to-one  corre¬ 
spondence  between  risk  factors  and  individual  indicators. 


1  10  U.S.C.  §2432  outlines  the  requirements  for  SAR  submissions,  which  are  mandatory  for  all  ACAT  I  pro¬ 

grams.  Program  managers,  through  the  DAMIR  SAR  module  application,  prepare  SARs.  Quarterly  reports 

are  submitted  as  major  changes  are  observed  in  programs.  SARs  are  annual  unless  reporting  an  APB  or  Nunn- 

McCurdy  breach,  in  which  case  a  program  submits  a  quarterly  SAR  (in  April,  June,  and  September).  Both  SAR 
and  DAES  reporting  begin  at  Milestone  B,  unless  otherwise  initiated  by  the  MDA. 

5  DAES  reporting  usually  begins  once  a  program  is  initiated  at  Milestone  B  and  ends  after  the  program  submits 
its  final  SAR.  DAES  is  reported  quarterly  in  three  groups  (A,  B,  and  C). 
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Table  8.5 

Candidate  Types  of  Risk  and  Related  Metrics 


Risk  Dimension 

Definition 

Potential  Metric 

Technical 

Risk  that  originates  from  technology, 
development,  integration,  testing, 
or  maturity  issues 

TRL  levels 

Growth  in  RDT&E  spending 

KPPs/KSAs  met  or  unmet 

Testing  failures 

Political  (external) 

Risk  that  originates  from  political 
attention  or  events  that  may  affect 
program  funding,  oversight,  etc. 

Nunn-McCurdy  or  APB  breaches 

Program  size 

Congressional  mentions  (testimony) 

Industrial  base  dispersion  (i.e.,  how 
many  states/districts  is  it  produced  in) 

Schedule  changes 

Administration  changes 

Military  action/operations 

Policy  (internal) 

Risk  that  results  from  policy  changes 
affecting  reporting  requirements, 
program  oversight 

Program  size  (in  dollars) 

Number  of  DAB-  or  MDA-level 
meetings3 

Number  of  acquisition  documents 
required 

Number  of  waivers  requested 

Cost 

Risk  that  results  from  cost  growth, 
both  unit  cost  growth  and  total  cost 
growth 

Unit  cost  growth/total  cost  growth 
Quantity  changes 

Funding 

Risk  that  relates  to  the  stability 
or  amount  of  funding  (as  well  as 
budget  sufficiency) 

Government  budget  changes 
Commercial/industry  economic  changes 
Budget  sufficiency  (do  funds  cover 
expected  cost?) 

Percentage  of  funds  remaining 

Requirements 

Risk  related  to  the  possibility  that 
the  program  will  not  meet  its 
current  or  future  requirements 
and  therefore  not  deliver  needed 
capabilities 

Schedule  delays  in  operational  or 
developmental  testing 

KPPs/KSAs  met  or  unmet 

Sustainment  costs 

Age/maturity  of  program 

a  DABs  are  the  senior  advisory  boards  for  defense  acquisitions  within  DoD  and  are  responsible  for 
approving  new  MDAPs  and  for  reviewing  MDAP  performance. 


After  reviewing  all  of  these  potential  metrics,  we  selected  a  smaller  set  of  descrip¬ 
tive  program-level  metrics,  listed  in  Table  8.6,  to  apply  to  our  test  portfolio  based  on 
the  following  criteria: 

•  Potential  usefulness  to  decisionmakers:  We  considered  the  types  of  informa¬ 
tion  provided  by  each  metric  and  aimed  to  include  those  metrics  that  provided 
the  most  valuable  insight  to  policymakers  about  the  status  of  the  overall  portfolio 
on  multiple  dimensions. 
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Table  8.6 

Program-  and  Portfolio-Level  Metrics 


Program-Level  Metric 

Outcomes/Indicators 

Measured 

Portfolio  Metric 

Descriptive-Type  Metrics 

Size  of  program  (dollars) 

Cost  outcomes,  political 
(external)  and  policy  risk 

Average,  median,  standard  deviation  in 
dollar  value 

Percentage  of  funds 
remaining 

Cost  and  schedule  outcomes, 
overall  risk 

Average,  median,  standard  deviation 
in  percentage  of  funds  remaining 

Percentage  of  time 
remaining 

Cost  and  schedule  outcomes, 
overall  risk 

Average,  median,  standard  deviation 
in  percentage  of  time  remaining 

Latest  milestone  Achieved 

Cost  and  schedule  outcomes, 
overall  risk 

Percentage  of  programs  passed  Milestone  B, 
Milestone  C 

Percentage  change  in 
quantity 

Cost  outcomes, overall  risk 

Average,  median,  standard  deviation 

"Churn":  average,  median,  standard 
deviation  of  absolute  value  of  percentage 
change  in  quantity 

Percentage  of  programs  with  a  quantity 
change  over  the  previous  year 

Performance-Type  Metrics 

Nunn-McCurdy  or  APB 
breaches 

Cost,  schedule,  performance 
outcomes,  political  risk 

Number  of  new  breaches 

Cumulative  total  breaches 

Percentage  of  programs  with  at  least  one 
breach 

Unit  cost 

growth:  percentage  change 
in  APUC  or  PAUC  (current 
and/or  original  baseline) 

Cost  outcomes,  funding  risk 

Average,  median,  standard  deviation  in 
PAUC/APUC  growth  from  current  baseline 

Percentage  of  programs  with  increase 
in  percentage  unit  cost  growth 

Distribution  of  percentage  unit  cost  growth 

RDT&E  and  procurement 
cost  growth 

Cost  outcomes,  technical  and 
requirements  risk 

Average,  median,  standard  deviation  in  cost 
growth 

Percentage  of  KPP  at  or 
above  threshold3 

Performance  outcomes, 
technical  and  requirements 
risk 

Average,  median,  standard  deviation 
of  percentage  KPP  at  or  above  threshold 

a  Unable  to  collect  consistent  data. 


•  Consistency  in  reporting  across  programs:  We  aimed  to  collect  data  generally 
available  for  all  programs  and  all  years  of  the  analysis,  rather  than  metrics  that 
were  missing  for  a  large  number  of  programs  in  the  portfolios  under  consider¬ 
ation. 

•  Comparability  of  descriptive  metrics  across  programs:  We  considered  the 
meaning  and  calculation  of  metrics  across  programs  and  years  and  chose  metrics 
that  provided  identical  types  of  information  across  programs  over  metrics  that 
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were  calculated  in  different  ways  or  had  different  meaning  for  different  programs 
or  in  different  years. 

•  Availability  of  data  to  the  RAND  team  and  decisionmakers:  We  tried  to 
choose  data  sources  that  were  unclassified,  that  were  easily  accessible  using  stan¬ 
dard  databases,  and  that  did  not  require  special  access. 

•  Ease  of  gathering  and  recording  data:  We  considered  the  difficulty  of  data 
collection,  the  number  of  different  documents  and  sources  that  needed  to  be 
obtained  or  consulted,  and  the  number  of  transformations  or  calculations  needed 
to  get  the  data  into  usable  form. 

These  criteria  limited  our  metrics  to  those  that  were  reported  in  or  could  be  calcu¬ 
lated  from  annual  or  quarterly  SARs  or  quarterly  DAES  reports.  We  found  that  these 
reports  had  the  most  consistent  and  reliable  reporting  formats  over  time  and  between 
programs  and  were  accessible  to  all  analysts  with  access  to  DAMIR.  In  determining 
potential  usefulness  to  decisionmakers,  we  focused  on  data  that  reported  cost  and 
schedule  performance  outcomes  for  the  program  or  provided  an  indicator  of  risk  expo¬ 
sure  (technical,  cost,  policy,  or  overall  risk). 

In  our  original  framework,  described  at  the  start  of  this  chapter,  we  developed 
portfolio  metrics  by  rolling  up  program  metrics  and  aggregating  them  to  describe  the 
overall  status  of  an  entire  set  of  programs,  rather  than  the  status  of  a  single  program. 
For  example,  although  percentage  change  in  unit  cost  is  a  metric  that  we  would  want 
to  observe  at  the  program  level,  at  the  portfolio  level,  we  are  interested  in  such  metrics 
as  the  mean  and  median  in  percentage  change  in  unit  cost  across  a  set  of  programs. 
Many  past  performance  assessments  focus  more  exclusively  on  program-level  metrics 
and  fail  to  take  the  larger,  portfolio  view  into  account.  However,  a  notable  exception 
is  DAMIR’s  relatively  new  “portfolio  view,”  which  provides  snapshot  views  of  several 
metrics  across  a  variety  of  defined  portfolios.  Table  8.7  provides  a  summary  of  this  tool 
within  DAMIR. 

Table  8.7 

Portfolio  Views  Within  DAMIR 


Portfolio 


Dashboard 

(SAR  and  DAES  Data  Visualization) 
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Contractors 
Contract  location 
Contract  performance 
Current  estimate  by  appropriation 
Current  cost  variance  changes  by  appropriation 
Funding  overview  by  service 
Funding  projection  total  by  appropriation 
Total  funding  by  commodity 
Total  funding  by  program 
Program  with  most  funds  remaining 
Program  phase  by  milestones 
Program  status  by  service 
Sunk  cost 

Unit  cost  by  percentage  change 
By  explicitly  defining  and  focusing  on  portfolio  metrics,  our  methodology  shifts 
to  a  higher  level  of  analysis  and  provides  a  different  perspective  on  portfolio  assessment. 
In  developing  the  portfolio-level  metrics,  listed  in  Table  8.6,  we  also  considered  the 
types  of  assessments  that  these  metrics  might  be  used  for.  The  selected  metrics  allow 
for  the  following  types  of  assessments: 

•  assessment  of  portfolio  status  on  a  single  metric 

•  integrated  assessment  of  portfolio  status  on  multiple  metrics 

•  a  static  “point-in-time”  assessment  of  a  portfolio  or  subportfolio 

•  comparisons  of  portfolio  or  subportfolio  status  over  time  (for  year-on-year,  or 
longer-term  trends) 

•  comparisons  of  portfolio  composition. 

The  data  we  collected  fit  our  criteria  for  consistent  reporting,  comparability  across 
portfolios,  and  availability  and  ease  of  collection.  In  addition,  the  resulting  metrics 
each  provided  specific  information  useful  to  policymakers  when  surveying  a  single 
program  as  well  as  a  larger  portfolio.  Although  the  metrics  by  themselves  can  provide 
useful  information,  it  is  when  these  are  integrated  that  a  policymaker  receives  a  more 
nuanced  view  of  the  portfolio.  We  describe  each  metric  below,  as  well  as  key  relation¬ 
ships  between  them. 

The  metrics  selected  fall  into  two  major,  interrelated  categories:  descriptive  and 
performance.  Descriptive  metrics  that  outline  characteristics  of  a  program  or  portfolio 
help  provide  context  for  the  performance  metrics.  For  example,  we  considered  program 
size,  in  terms  of  total  program  value  or  cost.  Very  large  programs  in  terms  of  total  cost 
(e.g.,  Joint  Strike  Fighter  [JSF])  may  have  more  political  importance,  visibility,  and 
therefore  more  “weight”  within  the  portfolio  as  a  whole.  In  addition,  percentage  unit 
cost  growth  alone  does  not  provide  enough  information  to  understand  the  relative 
influence  of  different  programs  on  the  portfolio.  Cost  growth  of  5  percent  for  a  pro¬ 
gram  such  as  the  JSF  with  a  total  program  cost  upward  of  $270  billion  would  affect 


Services  (Army,  Air  Force,  Navy)  and  DoD 
DAES  Group  (A,  B,  C) 

Acquisition  program  (C3I,  MAIS,  MDAP) 

Status  (active,  inactive,  all) 

Type  (aircraft:  bomber,  C3I,  fighter,  other,  transport, 
UAS;  C3I;  combat  vehicle;  helicopter;  missile; 
munitions;  radar;  satellite;  ship;  submarine; 
transport  vehicle) 
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the  overall  status  of  the  portfolio  more  than  the  same  percentage  cost  growth  for  a  $17 
billion  program  such  as  Excalibur.  To  account  for  this,  we  included  metrics  for  total 
program  cost  at  the  program  level;  and  at  the  portfolio  level,  we  examined  the  aver¬ 
age,  median,  and  standard  deviation  of  total  program  cost.  Finally,  larger  programs 
in  terms  of  total  cost  may  also  garner  more  political  or  external  scrutiny  leading  to  a 
higher  level  of  program  risk. 

Other  descriptive-based  metrics  provide  insight  into  a  program  or  portfolio’s 
overall  exposure  to  risk.  In  this  vein,  we  examined  four  metrics:  percentage  of  funds 
remaining,  percentage  of  time  remaining,  latest  milestone  achieved,  and  percentage 
change  in  quantity.  Changes  in  portfolio  median  for  the  percentage  of  funds  and  time 
remaining  may  indicate  portfolio  maturity  and  stability.  One  might  assume  that  a 
more  mature  portfolio  could  have  a  lower  level  of  future  risk  as  a  result  of  program 
stability  and  development.  A  similar  observation  can  be  made  about  the  percentage  of 
programs  in  a  portfolio  that  have  passed  Milestone  C.  Before  Milestone  C,  programs 
are  still  in  the  development  and  testing  phase — the  phase  when  most  of  the  techni¬ 
cal,  integration,  and  manufacturing  issues  that  can  affect  program  cost  and  schedule 
arise.  After  Milestone  C,  programs  typically  entered  LRIP,6  and  overall  program  risk 
type  changes  from  primarily  technological  or  developmental  to  primarily  production 
or  production-related.  When  normalizing  for  quantity  changes  in  procurement,  most 
studies  find  that  procurement  cost  growth  is  lower  than  development  cost  growth.7  A 
greater  percentage  of  post-Milestone  C  programs  indicate  a  more  mature  portfolio. 

Finally,  changes  in  quantity  are  an  indicator  of  program  stability.  Farge  or  fre¬ 
quent  quantity  changes  in  either  the  positive  or  negative  direction  can  be  an  indicator 
of  program  performance,  political  support,  fluctuating  operator  requirements,  or  other 
program  issues.  Instability  in  quantity  can  also  affect  cost  growth  risk;  prior  analy¬ 
sis  has  found  that  up  to  21.9  percent  of  cost  growth  for  MDAPs  can  be  explained  by 
program  quantity  changes.8  At  the  portfolio  level,  measuring  the  size  and  frequency 
of  program  quantity  changes  can  indicate  portfolio  stability.  Flowever,  it  is  also  worth 
noting  that  changes  in  quantity  are  not,  in  and  of  themselves,  indicators  of  a  problem 
within  the  affected  program.  For  example,  a  large  increase  in  quantity  for  the  Toma¬ 
hawk  missile  in  2011  reflected  not  a  program  failure  but  instead  an  increase  in  demand 
during  the  Fibya  crisis.9  This  observation  is  an  important  one  that  also  applies  to  other 
metrics.  In  many  cases,  it  is  difficult  to  interpret,  out  of  context,  whether  the  value  of  a 
given  metric  is  a  sign  of  increased  risk,  problems  within  a  given  program,  or  an  external 


6  LRIP  can  be  entered  before  Milestone  C  in  some  cases  and  is  not  always  mandatory. 

7  As  described  in  Joseph  G.  Bolten,  Robert  S.  Leonard,  Mark  V.  Arena,  Obaid  Younossi,  and  Jerry  M.  Sollinger, 
Sources  of  Weapon  System  Cost  Growth:  Analysis  of  35  Major  Defense  Acquisition  Programs,  Santa  Monica,  Calif.: 
RAND  Corporation,  MG-670-AF,  2008,  p.  xvi. 

8  Bolten  et  al.,  2008 

9  “Raytheon’s  Tomahawk  in  Demand,”  Zacks  Equity  Research,  June  11,  2012. 
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factor  affecting  the  program.  Instead,  most  metrics  flag  possible  problems  that  require 
additional  investigation  before  they  are  interpreted  one  way  or  the  other. 

As  stated,  the  aforementioned  metrics  provide  a  degree  of  context  for  the  more 
performance-based  metrics,  which  focus  on  schedule  and  cost  growth  status  or  out¬ 
comes.  Cost  growth  is  arguably  one  of  the  most  closely  monitored  performance  metrics 
by  managers  as  an  indicator  of  program  success  or  failure.  It  can  be  driven  by  a  range 
of  factors,  many  of  which  relate  directly  to  the  risks  programs  are  managing,  includ¬ 
ing  technical  difficulty,  underestimation  of  technological  maturity  or  system  integra¬ 
tion  challenges,  and  poor  estimation  of  cost  inputs  (labor,  materials,  infrastructure 
needs),  among  other  factors.  A  brief  discussion  of  the  interpretation  of  cost  metrics  at 
the  program  and  portfolio  levels  provides  insight  into  how  program  metrics,  portfolio 
metrics,  and  portfolio  risk  and  status  fit  together.  In  general,  upward  trends  in  program 
or  portfolio  unit  cost  metrics  over  time  can  be  an  indicator  of  increased  portfolio  risk. 
Cost  growth  in  specific  programs  within  a  portfolio  affects  the  overall  status  of  that 
portfolio  and  provide  an  indication  of  risk  within  the  portfolio — risk  of  additional  cost 
growth,  possible  cost  or  schedule  breaches,  possible  funding  issues,  and  other  possible 
concerns.  Cost  growth  at  the  portfolio  level,  then,  signals  policymakers  and  acquisition 
executives  that  additional  inquiry  is  required  to  diagnose  the  source  and  meaning  of 
any  possible  problem. 

For  our  unit  cost  metrics  at  the  program  level,  we  chose  to  focus  on  the  current 
baseline  metrics  for  PAUC  and  APUC  because  the  original  baselines  were  not  consis¬ 
tently  reported  in  the  SARs  over  the  time  period  2002  to  2012. 10  In  addition,  the  cur¬ 
rent  baseline  reflects  any  changes  in  the  program  that  required  the  establishment  of  a 
new  APB.  In  this  regard,  we  made  the  assumption  that  if  a  program  had  rebaselined, 
then  issues  that  affected  cost  and  schedule  risk  would  be  addressed  at  that  time  and  the 
current  baseline  would  reflect  the  current  status  of  the  program.  At  the  portfolio  level, 
we  examined  the  average,  median,  and  standard  deviation  of  unit  cost  growth  over 
time  and  tested  for  statistical  significance.* 11  We  also  examined  the  percentage  of  pro¬ 
grams  with  increases  in  unit  cost  growth  and  the  distribution  of  cost  growth  to  provide 
an  indication  of  the  variation  in  cost  performance  within  the  portfolio. 


10  Statutorily  mandated  reporting  of  the  original  baseline  in  the  SARs  did  not  begin  until  2006,  and  at  that 
point,  some  programs  reported  the  current  baseline  as  the  original  baseline  and  some  programs  reported  the 
original  APB  as  the  original  baseline.  Additional  discussion  on  the  issue  and  effects  of  rebaselining  can  be  found 
in  the  next  section  and  in  more  detail  in  Mark  V.  Arena,  Irv  Blickstein,  Daniel  Gonzales,  Sarah  Harting,  Jen¬ 
nifer  Lamping  Lewis,  Michael  McGee,  Megan  McKernan,  Charles  Nemfakos,  Jan  Osburg,  Rena  Rudavsky,  and 
Jerry  M.  Sollinger,  DoD  and  Commercial  Advanced  Waveform  Developments  and  Programs  with  Multiple  Nunn- 
McCurdy  Breaches,  Volume  5,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-1171/5-OSD,  2014a,  pp.  101—105. 

11  The  average  and  median  are  measures  of  central  tendency,  that  is,  the  values  where  most  of  the  data  appear 
to  be  clustered.  The  average  is  equal  to  the  sum  of  all  observations  divided  by  the  number  of  observations,  and 
the  median  in  the  value  exactly  in  the  middle  of  the  distribution.  The  standard  deviation  measures  the  spread  or 
dispersion  of  the  data.  Statistical  significance  is  a  measure  of  how  meaningful  a  given  set  of  results  is.  It  assesses 
whether  the  observed  results  or  values  are  likely  to  be  due  to  chance  or  represent  a  meaningful  trend  in  the  data. 
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We  also  examined  total  RDT&E  and  procurement  cost  growth  as  components 
of  total  cost  growth.  The  RDT&E  cost  growth  component  was  selected  as  an  indica¬ 
tor  of  technical  risk  in  the  program;  unanticipated  increases  in  development  costs  are 
often  due  to  technical  factors  or  technology  requirements  instability.  An  increase  in  the 
portfolio  median  for  this  metric  might  indicate  technical  or  integration  issues,  or  such 
nontechnical  issues  as  requirements  instability  or  inaccurate  cost-estimating  assump¬ 
tions.12  We  would  expect  RDT&E  cost  growth  to  be  higher  in  portfolios  with  more 
pre-Milestone  C  programs  or  those  that  are  less  mature  (i.e.,  fewer  average  years  past 
Milestone  B).  Procurement  costs  include  the  recurring  and  nonrecurring  costs  associ¬ 
ated  with  the  production  and  initial  support  of  the  system. 

Commonly  used  metrics — Nunn-McCurdy  and  APB  breaches — are  often  lag¬ 
ging  program  indicators,  that  is,  breaches  often  occur  after  the  program  has  encoun¬ 
tered  problems.  A  program  may  have  a  significant  or  a  critical  Nunn-McCurdy  breach 
in  APUC  or  PAUC.  These  breaches  are  measured  also  against  the  original  and  current 
baselines.13  In  addition,  in  any  given  year,  a  program  may  have  an  APB  breach  in  one 
or  more  of  eight  categories.14  These  breaches  trigger  a  series  of  review,  reporting,  and 
certification  activities  that  may  result  in  program  redirection,  restructure,  or  termina¬ 
tion.  At  the  portfolio  level,  the  percentage  of  programs  that  have  breaches  can  signify 
the  status  of  the  portfolio  in  the  given  year. 

In  addition  to  collecting  information  on  cost  and  schedule  growth,  we  attempted 
to  collect  consistent  data  on  the  percentage  of  key  performance  parameters  (KPPs)  that 
were  at  or  above  threshold  as  an  indicator  of  portfolio  performance  and  technical  risk. 
However,  data  on  KPPs  were  not  consistently  reported  (in  SARs  and  DAES)  across  ser¬ 
vices  and  programs,  and  thus  we  were  unable  to  develop  a  valid  indicator  at  the  port¬ 
folio  level.  Also,  past  RAND  research  found  performance  metrics  to  be  “insufficient 
proxies”  of  technical  risk  and  difficulty  as  certain  performance  indicators  may  carry 
more  influence  than  others  or  the  sample  may  contain  incomparable,  unlike  items.15 


12  Further  details  on  these  issues  can  be  found  in  Bolten  et  ah,  2008;  and  Obaid  Younossi,  Mark  V.  Arena, 
Robert  S.  Leonard,  Charles  Robert  Roll,  Jr.,  Arvind  K.  Jain,  and  Jerry  M.  Sollinger,  Is  Weapon  System  Cost  Growth 
Increasing?  A  Quantitative  Assessment  of  Completed  and  Ongoing  Programs,  Santa  Monica,  Calif.:  RAND  Corpora¬ 
tion,  MG-588-AF,  2007. 

13  A  “significant”  breach  occurs  when  APUC  or  PAUC  increases  15  percent  or  more  over  the  current  baseline 
estimate  or  30  percent  or  more  over  the  original  baseline  estimate.  A  “critical”  breach  occurs  when  APUC  or 
PAUC  increases  25  percent  or  more  over  the  current  baseline  estimate  and  50  percent  or  more  over  the  original 
baseline  estimate. 

14  These  APB  categories  are  schedule,  performance,  RDT&E  cost,  procurement  cost,  military  construction 
(MILCON)  cost,  O&S  cost,  PAUC,  and  APUC.  APB  breaches  occur  when  a  program  exceeds  the  baselines  set 
in  the  APB  document,  which  defines  the  targets  for  cost  growth,  performance,  and  schedule.  Specific  definitions 
are  provided  in  Chapter  Nine. 

15  See  Jeffrey  A.  Drezner,  Jeanne  M.  Jarvaise,  Ron  bless,  Daniel  M.  Norton,  and  Paul  G.  Hough,  An  Analysis  of 
Weapon  System  Cost  Growth,  Santa  Monica,  Calif.:  RAND  Corporation,  MR-291-AF,  1993.  See  also  Edmund 
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As  noted  in  the  previous  chapter,  the  set  of  metrics  we  include  in  this  methodology 
is  only  one  of  many  possible  sets  that  could  be  used  to  assess  portfolio  performance.  We 
intentionally  chose  about  ten  metrics  that  cover  key  aspects  of  portfolio  performance, 
believing  that  this  number  can  provide  diverse  and  varied  insights  without  becoming 
overly  complicated  and  complex  for  analysts  and  decisionmakers.  Analysts  may  choose 
to  focus  on  a  smaller  subset  of  these  metrics  when  it  comes  time  to  actually  present 
insights  to  policymakers  or  may  incorporate  other  metrics  that  address  specific  ques¬ 
tions  of  interest.  Also  stated  above,  one  must  understand  the  interrelationships  among 
the  metrics  to  create  a  more  holistic  picture  of  the  portfolio  and  adequately  explain 
observed  patterns  and  trends.  Taken  together,  multiple  metrics  provide  context  and 
clues  as  to  the  root  causes  of  unusual  or  interesting  trends. 


Addressing  Data  Anomalies  and  Challenges 

While  collecting  data,  we  came  across  a  number  of  data  anomalies,  including  missing 
data,  programs  that  split  or  changed  with  the  addition  of  new  blocks  or  modifications, 
and  programs  that  rebaselined  during  the  assessment  period  (affecting  unit  cost  and 
quantity  calculations).  Many  of  these  issues  arose  during  the  test  case  portfolio  data 
collection,  and  the  assumptions  made  to  address  those  were  carried  over  to  the  helicop¬ 
ter  and  satellite  portfolio  data  collection.  To  maintain  an  auditable  product,  analysts 
performing  their  own  portfolio  analysis  will  need  to  carefully  track  these  anomalies 
and  the  assumptions  or  measures  they  took  to  address  these.  To  describe  the  data 
anomalies  and  how  we  addressed  them,  we  provide  here  a  short  background  on  the 
programs  included  in  the  test  case  portfolio  and  the  data  issues  that  arose. 

Our  test  case  drew  on  previous  RAND  research  for  OSD  that  had  developed 
an  approach  to  anticipating  Nunn-McCurdy  breaches.16  This  approach  reduced  a  list 
of  approximately  100  MDAPs  into  a  more  manageable  “watch  list”  of  programs  that 
should  be  monitored  more  closely  for  possible  breaches.  We  used  this  watch  list  as  our 
test  case  to  identify  important  data  issues  and  challenges  that  analysts  would  need  to 
overcome  to  complete  a  portfolio  assessment  (see  Table  8. 8). 17 


Dews,  Giles  K.  Smith,  Allen  A.  Barbour,  Elwyn  D.  Harris,  and  M.  A.  Hesse,  Acquisition  Policy  Effectiveness: 
Department  of  Defense  Experience  in  the  1970s,  Santa  Monica,  Calif.:  RAND  Corporation,  R-2516-DRE,  1979. 

16  Arena  et  al.,  2014b. 

17  We  used  this  as  the  test  case  for  two  primary  reasons.  First,  many  of  the  data  had  already  been  collected,  allow¬ 
ing  us  to  focus  on  the  methodology  not  the  data  collection.  Second,  this  portfolio  included  a  variety  of  programs 
that  are  representative  of  all  of  the  services  and  different  commodity  types.  This  allowed  us  to  see  if  there  were 
consistencies  in  how  data  were  reported  across  services  and  commodities  and  to  identify  data  issues  that  might 
affect  how  portfolio  metrics  are  reported.  Finally,  these  programs  were  on  the  “watch  list”  because  they  appeared 
more  likely  than  other  MDAPs  to  experience  a  breach  on  more  dimensions.  For  example,  some  had  experienced 
a  previous  APB  or  Nunn-McCurdy  breach.  Others  had  periods  of  high  and  rising  cost  growth  or  had  experi¬ 
enced  significant  production  delays.  As  a  result,  the  programs  in  our  test  case  portfolio  could  be  considered  “high 
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Table  8.8 

Initial  Test  Case  Programs;  2010  Watch  List 


Joint  Program 

Airborne  and  Maritime/Fixed  Station  (AMF)  Joint  Tactical  Radio  System  (JTRS) 

Chemical  Demilitarization-Assembled  Chemical  Weapons  Alternatives  (Chem  Demil-ACWA) 

Handheld,  Manpack  and  Small  Form  Fit  (HMS)  JTRS 
JSF 

Joint  Tactical  Networks  (JTN) 

Air  Force  Program 

AEHF 

Advanced  Medium-Range  Air-to-Air  Missile  (AMRAAM)  Air  Launch  Aerial-Intercept  Guided  Missile  90 
(AIM-90) 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals  (FAB-T)  Increment  1 
Global  Hawk  (RQ-4) 

GPS  MIA 

Joint  Air-to-Surface  Standoff  Missile  (JASSM)  (Baseline) 

NAVSTAR  GPS 
Reaper  (MQ-9) 

SBIRS-High 

WGS 


Army  Program 

Integrated  Air  and  Missile  Defense  (IAMD) 

Excalibur 

Joint  Land  Attack  Cruise  Missile  Defense  Elevated  Netted  Sensor  System  (JLENS) 
Patriot  (PAC-3) 

Navy  Program 

AIM-9X 

CH-53K 

Guided  Missile  Destroyer  (DDG)  100 
E-2D  Advanced  Hawkeye  (AHE) 

H-1  Upgrades 

Littoral  Combat  Ship  (LCS) 

Landing  Helicopter  Assault  (LHA)  6 
Landing  Platform  Dock  (LPD)  17  Class 
MH-60S 

Remote  Minehunting  System  (RMS) 

Tomahawk 

Virginia-class  (SSN  774) 

Vertical  Takeoff  and  Landing  Tactical  Unmanned  Air  Vehicle  (VTUAV) 

SOURCE:  Arena  et  al.,  2014b. 


risk”  relative  to  other  MDAPs.  By  selecting  this  test  portfolio,  we  intended  to  observe  how  our  selected  metrics 
reflected  this  portfolio  risk. 
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Data  Anomalies  and  Challenges  Within  the  Test  Case,  "Watch-List"  Portfolio 

Although  the  test  case  portfolio  is  covered  in  more  detail  in  Appendix  B,  we  discuss 
here  some  of  the  data  challenges  we  faced  during  our  test  case  analysis  to  make  some 
broader  observations  about  the  challenges  and  considerations  analysts  may  face  during 
a  portfolio  analysis.  As  noted  above,  the  watch  list  portfolio  consisted  of  32  programs 
active  in  2010  and  2012  and  represented  a  variety  of  commodity  types  across  the 
services.18  We  discovered  during  data  collection  that  certain  metrics  applied  differ¬ 
ently,  or  not  at  all,  across  commodity  types.  This  is  likely  to  be  true  for  any  portfolio  of 
programs  and  is  an  important  factor  that  must  be  taken  into  account  during  the  port¬ 
folio  analysis  process.  In  each  case,  these  anomalies  must  be  dealt  with  program-by¬ 
program,  and  analysts  may  also  need  to  consider  alternative  metrics  that  provide  insight 
into  the  status  of  nontraditional  programs.  For  example,  Chem-Demil-ACWA  is  essen¬ 
tially  a  service,  not  a  physical  product,  thus  precluding  the  use  of  metrics  derived  from 
quantities.  It  also  does  not  follow  a  typical  acquisition  schedule  path,  so  its  scheduling 
data  would  not  be  congruent  with  other  MDAPs  within  the  portfolio.  The  JTN  and 
the  NAVSTAR  GPS  User  Equipment  programs  also  did  not  include  quantities,  inhib¬ 
iting  the  calculation  of  metrics  that  depended  on  this  variable  (for  example,  unit  cost). 
In  addition,  many  of  the  programs’  SARs,  especially  the  satellite  programs,  did  not 
explicitly  state  estimated  milestone  achievement  dates  within  the  schedule  sections.  In 
these  cases,  the  team  recorded  milestone  achievements  based  on  the  acquisition  activi¬ 
ties  completed  by  the  SAR’s  date,  such  as  contract  awards  and  engineering  or  produc¬ 
tion  decisions. 

The  fact  that  our  portfolio  compared  multiple  years  presented  additional  chal¬ 
lenges  as  programs  introduced  new  increments  or  blocks,  split  into  separate  programs, 
were  cancelled,  or  completed  their  SAR  reporting.  These  challenges  are  almost  certain 
to  emerge  in  any  portfolio  analysis.  In  dealing  with  these  types  of  program  and  port¬ 
folio  changes,  the  most  important  considerations  are  consistency  and  transparency. 
Analysts  should  make  explicit  and  clear  the  ways  in  which  they  choose  to  treat  each 
program  and  be  sure  to  apply  a  consistent  set  of  rules  to  all  programs  within  the  port¬ 
folio.  For  instance,  the  AEHF  program  was  a  single  block  under  one  program  number 
(PNO)  in  2010;  however,  in  2012,  the  program  comprised  two  blocks  (SV  1-4  and  SV 
5-6)  with  disaggregated  data,  although  under  the  same  PNO.  AIM-9X  Block  I,  under 
PNO  581,  ended  in  2011  and  was  followed  by  AIM-9X  Block  II,  PNO  442,  in  2012. 
A  summary  of  all  of  these  issues  found  during  the  watch  list  portfolio  data  collection 
is  in  Table  8.9.  We  discuss  the  assumptions  made  to  address  these  challenges  in  a  sub¬ 
sequent  section. 


18  It  is  worth  noting  that  the  number  of  programs  in  the  test  case  portfolio  is  likely  larger  than  optimal  for  this 
methodology  of  portfolio  assessment.  However,  we  chose  to  use  this  as  the  test  case  anyway  for  the  data  avail¬ 
ability,  range  of  different  programs,  and  “higher  risk”  profile  (see  footnote  17).  Because  the  test  case  was  used  as 
a  tool  for  methodology  development,  we  felt  that  this  decision  was  justified. 
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Table  8.9 

Summary  of  Test  Case  Portfolio  Data  Issues 


Issue 

Program  Affected 

Notes 

Program  begins/ends  during 
portfolio  time  frame 

AIM-9X  Blocks  1  &  II 

AIM-9X  Block  1  stops  reporting  by  2012 
because  it  is  over  90  percent  complete;  Block 

II  begins  reporting  in  2012 

Reaper  (MQ-9) 

Reaper  did  not  report  an  APB  in  2010 
because  it  had  just  transitioned  from  ACAT  II 
to  ACAT  1  because  of  increased  funding 

Program  split  into  separate 
blocks/increments  under 

Advanced  EHF 

Data  are  reported  separately  for  Satellite 
Vehicle  (SV)  1-4  and  SV  5-6  in  2012 

same  PNO 

JASSM 

Data  are  reported  separately  for  JASSM 
Baseline  and  JASSM-ER  in  2010  and  2012 

JSF 

Data  are  reported  separately  for  the  F-35 
aircraft  and  engine  in  2012 

NAVSTAR  GPS 

Data  are  reported  separately  for  satellite  and 
control  and  user  equipment  in  2010  and  2012 

SBIRS-High 

Data  are  reported  separately  for  the  Baseline 
and  Block  Buy  in  2012 

No  quantities  noted 

Chem-Demil-ACWA 

Nature  of  program  (service-based) 

JTN 

Nature  of  program  (software-based) 

No  milestone  noted 

Chem  Demil-ACWA 

Nature  of  program 

FAB-T 

Acquisition  activities  listed,  but  milestones 
not  explicitly  listed  in  schedule 

Global  Hawk 

Acquisition  activities  listed,  but  milestones 
not  explicitly  listed  in  schedule 

Data  Anomalies  and  Challenges  Within  the  Satellite  and  Helicopter  Portfolios 

Many  of  the  data  issues  that  emerged  in  the  test  case  portfolio  also  proved  problem¬ 
atic  in  the  satellite  and  helicopter  portfolios  (see  Table  8.10).  However,  we  faced  an 
additional  challenge  with  the  satellite  and  helicopter  portfolios,  because  of  changes  in 
portfolio  composition  over  the  11-year  time  frame  of  our  investigation.  In  our  analy¬ 
sis,  a  program  leaves  the  portfolio  when  it  is  completed  or  cancelled.  Defining  when  a 
program  enters  the  portfolio  is  somewhat  less  straightforward.  Because  SARs  were  the 
primary  data  source,  we  decided  to  define  “portfolio  entry”  as  when  a  program  first 
began  reporting  SARs.  This  is  typically  when  a  program  enters  Milestone  B,  although 
some  programs  begin  reporting  earlier  (e.g.,  DDG  1000,  F-35).  Because  programs  were 
constantly  moving  into  and  out  of  the  portfolio,  we  had  to  pay  more  careful  attention 
to  changes  in  composition  as  we  calculated  and  interpreted  portfolio  metrics.  These 
changes  in  composition  are  not  a  problem  per  se,  but  they  are  considerations  that  need 
to  be  taken  into  account  as  metrics  are  calculated  and  interpreted. 

The  second  data  issue  that  arose  during  our  data  collection  for  the  satellite  and 
helicopter  portfolios — one  that  was  not  addressed  during  the  test  case  analysis — 
involved  the  availability  of  SARs  in  2008.  We  discovered  that  DAMIR  had  very  few 
SARs  posted  for  that  year  and,  thus,  we  relied  on  the  DAES  most  equivalent  to  the 
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Table  8.10 

Data  Issues  in  Satellite  and  Helicopter  Portfolios 


Issue 

Program  Affected 

Notes 

Program  begins/ends 
during  portfolio  time 

CH-53K 

Began  reporting  in  2004 

frame 

Comanche 

Stopped  reporting  in  2004 

Longbow  Apache 

Stopped  reporting  in  2011 

ARH 

Began  reporting  in  2005,  stopped  reporting  in 
2009 

VH-71 

Began  reporting  in  2005,  stopped  reporting  in 
2009 

LUH 

Began  reporting  in  2006 

AH-64E  New  Build 

Began  reporting  in  2010 

AH-64E  Remanufacture 

Began  reporting  in  2006 

MUOS 

Began  reporting  in  2004 

GPS  III 

Began  reporting  in  2008 

GPS  OCX 

Began  reporting  in  2012 

NESP 

Stopped  reporting  in  2005 

NPOESS 

Stopped  reporting  in  2012 

SBSS  Block  10 

Began  reporting  in  2007,  stopped  reporting  in 
2011 

TSAT 

Reported  only  in  2004 

Program  split  into 

SBIRS-High 

Data  are  reported  separately  for  the  Baseline  and 

separate  blocks/ 

Block  Buy  in  2012 

increments  under  same 
PNO 

NAVSTAR  GPS 

Data  are  reported  separately  for  satellite  and 
control  and  user  equipment  in  2010  and  2012 

Longbow  Apache 

Data  are  reported  separately  for  airframe 
modifications  and  FCR  mission  kit  in  2002  and 

2003 

No  quantities  noted 

Not  applicable 

Not  applicable 

No  milestone  noted 

GPS  III 

Uses  "key  decision  points" 

MUOS 

Uses  "key  decision  points" 

NAVSTAR 

Reports  by  production  segments 

SBSS  Block  10 

Uses  "key  decision  points" 

TSAT 

Uses  "key  decision  points" 

No  2008  SARs 

All  except  H-1  Upgrades 

In  2008,  SARs  were  not  necessarily  submitted; 
instead,  PB2010  Limited  SARs  were  submitted, 
which  are  not  included  within  DAMIR 
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December  SAR.19  The  GAO  encountered  a  similar  problem  during  this  time  frame, 
stating  that  it  “did  not  analyze  the  cost  and  schedule  performance  of  the  [2008]  portfo¬ 
lio  because  DoD  did  not  issue  timely  or  complete  Selected  Acquisition  Reports  for  the 
second  consecutive  presidential  transition.”20  After  Congressional  protest,  the  admin¬ 
istration  submitted  “PB2010  Limited”  SARs;  however,  these  did  not  contain  the  same 
level  of  budgetary  information  and  were  not  considered  “official,”  so  they  are  not  avail¬ 
able  with  the  DAMIR  information  system.21 

Assumptions  Made  to  Address  Anomalies 

To  maintain  consistent  data  collection  and  cleaning,  we  relied  on  a  number  of  assump¬ 
tions  to  address  data  anomalies.  We  used  the  same  assumptions  across  the  test  case 
and  satellite  and  helicopter  portfolios.  These  “rules”  establish  a  repeatable  process  that 
allows  for  comparison  across  all  programs  and  portfolios  and  are  listed  in  Table  8.11. 

Finally,  in  addition  to  these  common  issues  and  mitigating  assumptions,  we  also 
had  to  develop  a  strategy  to  deal  with  unique  issues,  such  as  software  programs,  whose 
characteristics  make  certain  metrics  not  applicable.  For  these  programs,  we  simply 


Table  8.11 

Mitigating  Assumptions  to  Address  Data  Anomalies 


Common  Data  Issue 

Mitigating  Assumption 

Determining  when  to  begin  and  end  data 
collection  over  the  time  frame  of  portfolio 
analysis 

Begin  and  end  collection  of  data  based  on  when  the 
program  is  reporting  SARs 

Program  split  into  separate  blocks/increments/ 
components  under  same  PNO 

Combine  subprograms  and  components  into  a  single 
program,  collecting  cumulative  data  when  applicable 

Program  split  into  separate  blocks/increments 
under  different  PNOs 

Consider  blocks/increments/components  as  separate 
programs 

No  quantities  noted 

Record  data  entry  as  "non-applicable"  and  do  not 
include  program  in  that  metric's  analysis 

No  milestone  noted 

Using  listed  acquisition  activities  under  the  schedule, 
apply  DoDI  5000.02  guidelines  for  assigning  an 
appropriate  "milestone  achieved"  code  during  the 
year  of  data  collection 

Data  not  reported  in  SARs,  or  SARs  unavailable 
for  a  given  year 

Although  not  as  heavily  vetted  as  SARs,  the  DAES  offer 
another  source  for  many  of  the  metrics;  the  use  of  an 
alternative  source  should  be  noted 

Generally  missing  data  with  no  identifiable  In  some  cases,  interpolation  or  calculation  from 

alternative  sources  congruent  data  may  be  appropriate,  but  if  this  is  not 

possible,  the  data  entry  will  be  labeled  as  "missing" 
and  not  included  in  the  overall  analysis 

^  Different  programs  may  report  in  different  months,  so  we  had  to  use  DAES  reports  from  several  months 

(October,  November,  and  December). 

20  Government  Accountability  Office,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  Washing¬ 
ton,  D.C.,  GAO-10-388SP,  March  2010. 

21  From  discussions  with  DAMIR  personnel. 
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record  “not  applicable”  for  the  affected  metrics.  This  is  one  limitation  of  our  meth¬ 
odology  as  it  means  that  our  assessment  will  not  capture  the  performance  or  status  of 
these  programs.  To  address  this  limitation,  an  assessment  phase  of  the  analysis  would 
need  to  incorporate  additional  and  higher-level  metrics  that  capture  the  effect  of  these 
programs  on  the  portfolio.  It  is  important  to  note  that  these  types  of  program-specific 
issues  will  affect  any  portfolio  under  consideration.  Each  program  has  unique  charac¬ 
teristics,  some  of  which  make  certain  metrics  difficult  or  meaningless.  For  these  pro¬ 
grams,  we  apply  our  business  rules  as  consistently  as  possible  across  programs,  and  we 
make  all  assumptions  and  adjustment  explicitly,  allowing  a  reader  to  clearly  assess  how 
we  have  approached  the  analysis  to  disagree  with  or  modify  our  assumptions  if  desired. 

Additional  Data  Considerations 

Data  collection  for  the  portfolio  analysis  includes  not  only  recording  relevant  metrics 
and  addressing  the  data  issues  described  above  but  also  processing  the  data  to  ensure 
that  they  are  comparable  across  years  and  programs.  For  example,  data  concerning 
funds  require  conversion  to  constant  year  dollars  to  account  for  inflation.  In  addition, 
defense  acquisition  programs  often  rebaseline — a  process  that  can  confound  the  bench¬ 
mark  against  which  metrics  are  calculated.  Therefore,  we  collect  data  on  and  track  cost 
and  quantity  changes  based  on  the  original  baseline  as  well  as  the  new  baseline. 


Calculating  Portfolio  Metrics  and  Statistical  Significance 

After  collecting  our  selected  program  metrics,  the  next  step  was  to  calculate  the  aggre¬ 
gated  portfolio  metrics  described  above.  Analysts  will  need  to  consider  how  they  wish 
to  compile  and  package  the  data  collected  in  terms  of  the  portfolio  type  as  well  as  the 
defined  objectives  of  the  portfolio  analysis  itself.  As  noted  above,  the  portfolio  metrics 
we  selected  are  intended  not  to  summarize  the  status  of  a  single  program  but  to  provide 
a  higher-level  assessment  of  the  portfolio  based  on  the  status  of  the  programs  included 
in  that  portfolio.  Different  portfolio  metrics  provide  information  on  different  aspects 
of  the  portfolio.  Because  our  intention  was  to  develop  a  repeatable  methodology  that 
could  be  applied  to  portfolios  of  all  kinds,  we  spent  considerable  time  developing  a 
generalizable  computer  program  that  can  easily  calculate  portfolio  metrics.22  This  same 
program  can  be  used  to  calculate  metrics  for  the  satellite  and  helicopter  portfolios  and 
will  be  equally  applicable  to  other,  larger  portfolios  with  more  years  of  data. 

As  noted  in  Table  8.6,  the  portfolio  metrics  that  we  calculated  included  mean, 
median,  and  standard  deviations  of  such  quantities  as  percentage  unit  cost  growth, 
quantity  change,  total  program  value,  and  percentage  of  RDT&E  and  procurement 


22  The  generalized  computer  program  was  developed  in  STATA,  a  statistical  package  used  to  conduct  data 
analysis. 
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funds  remaining.  We  also  counted  the  number  and  types  of  breaches  and  the  number 
of  programs  past  key  points  in  development,  such  as  Milestones  B  and  C.  Each  of  these 
portfolio  metrics  gave  us  a  unique  window  on  the  status  of  the  portfolio.  Furthermore, 
these  metrics  can  be  combined  and  viewed  together  to  provide  a  more  holistic  view  of 
portfolio  status. 

In  addition  to  looking  at  static  portfolio  metrics  to  assess  the  status  of  our  test 
case  programs  in  each  year,  we  also  hoped  to  compare  across  years  to  understand  how 
the  portfolio  changed  over  time.  This  involved  not  only  calculating  metrics  but  also 
determining  whether  changes  we  observed  between  years  were  statistically  significant 
(meaning  that  we  can  say  with  a  given  level  of  confidence,  usually  95  percent,  that  the 
observed  change  is  statistically  different  from  zero  or  that  the  change  is  a  real  change 
in  the  trend  and  is  not  simply  noise  in  the  data)  and  substantively  meaningful  (or 
sizable  enough  to  suggest  a  meaningful  change  over  time).  To  address  statistical  sig¬ 
nificance  in  the  test  case  with  only  two  years  of  data,  we  relied  on  paired  t-tests  for 
unweighted  data.  This  made  sense  in  the  context  of  our  test  case  portfolio  in  which  all 
programs  are  the  same  in  the  two  years.  The  paired  t-test  takes  this  program  consis¬ 
tency  into  account.  The  t-test  compares  the  difference  between  the  means  of  the  two 
samples  (2010  and  2012)  to  the  variance  within  each  sample  independently,  taking 
into  account  the  number  of  observations.  When  this  ratio  value  is  large  (greater  than 
two),  then  the  difference  between  the  two  years  is  said  to  be  statistically  significant  and 
is  unlikely  to  be  due  to  chance. 

For  the  satellite  and  helicopter  portfolios,  however,  we  needed  a  different  way 
to  assess  statistical  significance  because  we  had  so  many  more  years  of  data.  In  these 
instances,  we  conducted  a  number  of  tests.  First,  we  used  a  fixed-effects  model,  which 
is  able  to  account  for  program-specific  characteristics  (essentially  includes  a  control 
variable  for  each  program  in  the  analysis).  The  fixed-effects  model  allows  us  to  deter¬ 
mine  whether  observed  differences  in  portfolio  performance  between  any  two  years 
are  statistically  significant,  meaning  that  we  can  say  with  some  certainty  that  they  are 
different  from  zero  or  that  the  change  is  a  real  change  in  the  trend  and  is  not  simply 
noise  in  the  data.  For  example,  we  can  use  the  results  of  the  model  to  determine  if  a 
difference  in  unit  cost  growth  between  2004  and  2005  is  statistically  meaningful  or 
likely  to  be  due  to  chance.  We  can  conduct  these  paired  comparisons  for  all  pairs  of 
years  and  for  each  metric.  We  can  also  use  the  overall  fixed-effects  coefficient  to  pro¬ 
vide  a  summary  measure  of  the  change  over  time,  on  average,  as  well  as  the  significance 
of  that  change. 

In  addition  to  statistical  significance,  we  are  also  interested  in  the  substantive 
importance  of  a  given  effect.  Even  if  an  effect  or  observed  change  is  statistically  signifi¬ 
cant,  we  may  not  care  about  that  change  if  it  is  substantively  small  or  insignificant.  For 
substantive  importance,  we  used  a  measure  known  as  Cohen’s  D,  which  also  compares 
the  difference  between  the  means  of  the  two  sample  years  and  the  differences  or  varia¬ 
tion  within  each  year.  When  this  value  is  large  (greater  than  0.7),  it  suggests  a  substan- 
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tively  meaningful  difference.  However,  when  the  value  is  very  small  (less  than  0.2)  the 
associated  effect  size  is  not  very  meaningful.  When  the  value  is  between  0.2  and  0.5, 
we  can  say  that  the  substantive  importance  is  small,  and  when  the  value  is  between  0.5 
and  0.7,  the  effect  size  is  moderate. 

In  addition  to  calculating  each  metric,  conducting  significance  tests,  and  con¬ 
sidering  substantive  importance,  we  also  conducted  some  additional  calculations  to 
address  anomalies  within  the  data  and  to  incorporate  program  size  as  a  measure  of  pro¬ 
gram  influence.  To  address  outliers,  we  conducted  all  analyses  both  with  and  without 
certain  programs  that  seemed  to  lie  far  from  the  rest  of  the  data.  The  best  examples  of 
this  occurred  in  our  test  case  portfolio,  where  we  calculated  the  total  program  value 
metrics  with  and  without  the  JSF  program  (the  largest  program  by  far)  and  the  PAUC 
metrics  (mean,  median,  and  standard  deviation)  with  and  without  JLENS.  Although 
we  do  not  want  to  ignore  these  programs  in  our  assessment,  we  also  want  to  under¬ 
stand  patterns  and  relationships  within  the  data  that  might  be  hidden  by  the  inclusion 
of  the  outlier. 

We  also  took  additional  steps  to  incorporate  program  size  into  our  calculation  of 
metrics,  calculating  all  the  metrics  using  both  unweighted  and  weighted  data,  employ¬ 
ing  program  value  as  the  relevant  weight.  By  doing  this,  we  address  the  fact  that  the 
status  of  the  largest  programs  in  the  portfolio  may  be  most  important  to  policymak¬ 
ers.  A  given  percentage  unit  cost  growth,  for  example,  will  be  most  significant  when  it 
occurs  in  a  large  program.  Similarly,  breaches  and  variability  in  quantity  may  be  most 
problematic  when  they  occur  in  large  programs. 


Visualizing  Portfolio  Metrics 

Calculations  of  metrics,  although  useful  for  the  analyst,  must  be  properly  pack¬ 
aged  in  a  way  that  is  digestible  for  the  policymaker.  Although  the  preferences  of  the 
policymaker  may  vary  from  Excel  sheets  to  narrative  forms,  data  visualizations  may  be 
necessary  to  quickly  indicate  implications  and  relationships  between  the  portfolios  and 
their  inputs.  To  analyze  the  portfolio  metrics  that  we  calculated,  we  developed  a  series 
of  visualizations  to  display  the  data.  Table  8.12  shows  a  list  of  the  visualizations  used 
and  the  information  that  each  is  intended  to  provide.  In  our  portfolio  analysis  of  the 
helicopter  and  satellite  portfolios,  we  present  all  of  these  visualizations  and  metrics  for 
each  portfolio.  However,  in  reality,  it  might  be  too  time  consuming  for  an  analyst  to 
develop  the  full  set  of  visualizations  for  all  portfolios  or  for  a  policymaker  to  digest  the 
information  presented  by  many  different  visualizations  for  many  different  portfolios. 
Using  Table  8.12,  however,  the  analyst  could  choose  to  complete  the  specific  visual¬ 
izations  that  address  policymaker  questions,  thus  tailoring  the  analysis  to  the  issues  of 
interest.  Furthermore,  it  is  worth  noting  that  these  visualizations  highlight  portfolio- 
level  trends.  Additional  analysis  might  be  required  to  identify  specific  programs  with 
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Table  8.12 

Visualizations  and  Information  Provided 


Visualization 

Type  of  Information  Provided 

Histograms  and  bar  charts 

Distribution  of  programs 

Trends  in  numerical  metrics 

Box-and-whisker  plots 

Medians  and  change  over  time 

Range  and  spread  of  data  (with  or  without  extreme  values) 

"Heat"  map 

Year-on-year  changes  across  the  portfolio  for  one  metric 

More  detailed  view  of  program-level  information 

Bubble  diagrams 

Two  metrics  displayed  simultaneously 

A  measure  of  a  program's  "influence"  in  portfolio 

One  way  to  visualize  a  weighted  metric 

Radar  charts 

Aggregate  view  of  several  different  metrics  at  one  or  several  points  in  time 

high  levels  of  risk  or  poor  performance.  Bar  charts  were  used  mainly  to  display  descrip¬ 
tive  information  about  the  portfolio — for  example,  number  of  programs  by  milestone, 
number  of  programs  by  service,  or  number  of  programs  by  DAMIR  category  (i.e. 
logistics,  force  application,  etc.).  These  bar  charts  allowed  us  to  analyze  portfolio  com¬ 
position  over  time. 

Box-and-whisker  plots  were  used  for  metrics  where  the  median  and  standard 
deviation  were  calculated;  these  metrics  included  unit  cost  and  program  funding  met¬ 
rics.  The  box-and-whisker  plots  display  the  statistical  distribution  of  the  data  including 
the  median,  the  degree  of  dispersion  (spread),  and  the  range  of  the  data.  An  example 
of  a  box-and-whisker  plot  for  the  test  (watch  list)  portfolio  is  displayed  in  Figure  8.1. 
The  box  represents  the  data  that  fall  within  the  25th  and  75th  percentiles  (the  middle 
50  percent  of  the  data)  called  the  interquartile  range  (IQR).  The  height  of  this  box 
represents  how  dispersed  those  data  are.  The  bars  that  extend  outside  of  the  boxes  are 
called  “whiskers,”  and  they  provide  a  measure  of  the  range  of  the  data,  connecting  the 
adjacent  values  1.5  times  the  IQR  above  and  below  the  median.  Data  points  that  lie 
outside  this  range  are  sometimes  represented  as  dots  but  are  sometimes  omitted  from 
the  graph  for  presentation  purposes.  Finally,  we  include  with  our  box-and-whisker 
plots  a  measure  of  effect  size,  using  Cohen’s  D  as  defined  above 

Another  visualization  technique  that  we  used  to  display  year-on-year  portfolio 
data  for  a  single  metric  is  a  “heat  map.”  A  heat  map  is  a  graphical  representation  of  data 
where  individual  values  or  ranges  of  values  are  represented  by  colors.  Fleat  maps  are 
useful  for  highlighting  key  trends  over  time.  In  many  of  these  charts,  a  “trend  to  green” 
suggests  a  general  improvement  in  performance  (where  red  is  poor  performance  and 
green  represents  good  performance).  We  applied  this  graphical  representation  to  the 
unit  cost  metrics  for  the  helicopter  and  satellite  portfolios  to  display  portfolio  perfor¬ 
mance  between  2002  and  2012.  Programs  with  no  shading  indicate  that  the  program 
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Figure  8.1 

Example  Box-and-Whisker  Plot;  Test  Portfolio  percentage  Change  in  Current  APUC,  2010 
and  2012 
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is  not  reporting  in  that  year  or  has  been  cancelled.  This  display  allows  the  decision¬ 
maker  to  make  comparisons  on  a  single  metric  across  years  (when  viewed  from  left  to 
right)  and  programs  (when  viewed  from  top  to  bottom). 

The  visual  displays  we  have  discussed  up  to  this  point  are  useful  for  displaying 
one  metric  over  time  or  across  programs.  However,  there  may  be  cases  where  we  would 
like  to  assess  program  performance  along  several  different  dimensions  simultaneously. 
We  found  that,  for  example,  in  our  display  of  PAUC  and  APUC  data,  we  were  able 
to  identify  poorly  performing  programs  within  the  portfolio,  but  we  were  missing  the 
substantive  influence,  or  relative  “weight,”  of  that  program  to  the  portfolio.  A  very 
large  program  (in  terms  of  total  program  cost)  that  is  performing  poorly  may  have 
more  of  a  negative  effect  on  overall  portfolio  status  than  a  relatively  small  program.  To 
indicate  the  relative  weight  of  programs  in  the  portfolio  we  used  bubble  chart  displays. 
Bubble  charts  are  used  to  show  relationships  between  three  variables.  The  area  of  the 
bubble  represents  the  relative  size  of  the  program  in  terms  of  estimated  total  program 
cost,  and  the  position  of  the  bubble  marks  the  APUC  and  PAUC  for  the  program  in  a 
given  year.  Programs  in  the  upper  right  quadrant  are  poorly  performing  programs  from 
a  cost  perspective,  whereas  programs  in  the  lower  left  quadrant  are  performing  better. 

Finally,  we  also  used  radar  charts  to  display  multiple  portfolio  metrics  on  one 
plot.  These  radar  charts  allow  the  user  to  get  an  aggregate  view  of  portfolio  status 
across  periods  of  time.  We  provide  examples  of  both  approaches  in  our  analysis  of  the 
helicopter  and  satellite  portfolios  in  subsequent  chapters. 
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It  is  possible  to  display  more  or  fewer  metrics  on  the  radar  depending  on  the 
portfolio  characteristics  that  the  decisionmaker  is  interested  in.  For  our  analysis,  we 
selected  six  metrics  that  offer  insight  into  portfolio  status.  Past  data  suggest  that  a 
larger  portion  of  a  program’s  cost  growth  occurs  in  the  pre-Milestone  C  phase;  thus,  if 
there  are  a  larger  percentage  of  post-Milestone  C  programs  in  the  portfolio,  the  port¬ 
folio  might  be  considered  lower  risk.  In  the  same  regard,  quantity  changes  (both  posi¬ 
tive  and  negative)  have  historically  had  an  effect  on  program  cost  growth;  therefore, 
the  more  programs  in  a  portfolio  with  quantity  changes  in  a  given  year,  the  less  stable 
a  portfolio  might  be.23  In  terms  of  APB  and  Nunn-McCurdy  breaches,  the  radar  plots 
indicate  the  percentage  of  programs  in  a  portfolio  with  any  breach  in  the  given  year. 
A  program  may  have  three  APB  breaches  in  one  year,  but  it  would  be  counted  only 
once  for  the  purpose  of  the  portfolio  display.24  Finally,  the  PAUC  and  APUC  metrics 
are  intended  to  capture  the  poorly  performing  programs  presented  in  the  heat  maps 
(programs  in  shades  of  red).  On  the  radar  plot,  we  indicate  the  percentage  of  programs 
in  the  portfolio  that  have  greater  than  5  percent  growth  in  PAUC  or  APUC  from  the 
current  baseline  in  the  given  year.  Some  caution  needs  to  be  taken  when  interpreting 
these  charts  from  year  to  year  on  a  single  dimension,  as  the  number  of  programs  in  the 
portfolio  often  changes  from  year  to  year.  Thus,  the  same  number  of  programs  may  be 
performing  poorly  from  year  to  year  but  the  shape  may  appear  to  shrink  if  a  new  pro¬ 
gram  is  added  that  is  performing  well.25 

Examples  and  interpretation  of  all  these  visualizations  will  be  provided  in  subse¬ 
quent  chapters. 


Chapter  Summary 

This  chapter  outlined  our  portfolio  analysis  framework,  which  included  the  following 
steps:  identify  objectives,  choose  portfolio  type,  select  data  and  metrics,  address  data 
anomalies  and  challenges,  calculate  metrics,  and  visualize  metrics.  In  selecting  port¬ 
folios  for  analysis,  we  focused  on  choosing  portfolios  that  would  offer  robust  tests  of 
our  methodology  and  would  allow  us  to  refine  these  methodologies  as  necessary.  In 
selecting  metrics,  we  chose  a  set  that  would  comprehensively  assess  program  and  port¬ 
folio  status  and  capture  many  cost  and  schedule  dimensions  of  program  and  portfolio 
risk  and  status.  We  also  discussed  the  visualizations  that  we  used  to  present  the  results 
of  our  analysis  and  provided  examples  of  these  visualizations  and  their  interpretation. 


23  Bolten  et  al.,  2008. 

24  This  is  a  function  of  how  we  have  defined  the  metrics  included  in  the  radar  plot. 

25  For  example  if  four  of  eight  (50  percent)  programs  have  APB  breaches  in  2010  and  then  a  new  program 
is  added  in  2011  that  has  no  APB  breaches,  the  radar  chart  will  appear  to  shrink  (44  percent)  on  the  APB 
dimension. 
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Finally,  we  used  a  set  of  visualizations  that  graphically  summarize  our  analysis  in  a  way 
that  may  be  useful  and  intuitive  for  policymakers.  In  the  next  chapter,  we  will  outline 
the  descriptive  metrics,  framed  by  our  initial  objectives  of  exploring  schedule  and  cost 
risk,  with  the  following  chapter  taking  a  deeper  dive  into  the  trends  and  patterns  of  the 
helicopter  portfolio’s  metrics  that  were  of  interest. 


CHAPTER  NINE 


Initial  Examination  of  Defined  Metrics:  Helicopter  and 
Satellite  Portfolios 


The  preceding  chapter  outlined  how  one  constructs  the  foundation  of  a  portfolio 
analysis.  In  this  chapter,  we  explore  in  greater  detail  how  one  then  assesses  a  set  of  met¬ 
rics  and  visualizations  through  the  lens  of  the  portfolio  analysis  objectives.  In  our  anal¬ 
ysis,  we  focused  primarily  on  cost  and  schedule  performance  indicators  as  the  primary 
objective  of  our  portfolio  view  of  helicopter  and  satellite  programs,  and  the  follow¬ 
ing  sections  will  outline  observations  from  that  perspective.  In  particular,  this  chapter 
will  explore  in  greater  depth  the  helicopter  and  satellite  example  portfolios  (we  discuss 
the  “watch  list”  test  case  portfolio  in  Appendix  B).  We  chose  to  discuss  the  portfo¬ 
lios  together  to  highlight  similarities  and  differences  in  the  metrics  and  visualizations 
across  the  portfolio  types.  Overall,  the  following  section  tells  the  “what”  of  the  portfo¬ 
lio  analysis:  What  is  the  composition,  or  characteristics,  and  the  performance  indica¬ 
tors  of  the  portfolio.  Chapter  Ten  provides  an  example  narrative  to  answer  potential 
policymakers’  follow-up  questions  as  to  the  “why”  and  “how”  the  portfolios  are  pre¬ 
senting  these  results. 


Descriptive  Metrics 

We  started  our  analysis  by  looking  at  the  descriptive  metrics:  latest  milestone  achieved, 
size  of  program,  percentage  of  funds  and  time  remaining,  and  percentage  change  in 
quantity,  or  churn.  Because  we  were  analyzing  not  just  a  snapshot  in  time  but  an 
11-year  time  frame,  we  must  also  note  how  the  composition  of  the  portfolio  changed 
over  time  as  programs  entered  and  left  the  portfolio.1  The  following  discussion  displays 


1  For  our  defined  portfolio,  programs  “entered”  or  “left”  the  portfolio  based  on  when  they  were  submitting 
SARs.  Therefore,  a  program  enters  the  portfolio  when  it  submits  its  first  SAR  and  leaves  the  portfolio  when  it 
ceases  SAR  submissions,  which  can  result  when  the  program  is  90  percent  complete  or  has  been  cancelled.  Port¬ 
folio  composition  also  entails  subcategories  of  the  portfolio  type  in  question.  For  example,  with  the  helicopter 
portfolio,  we  may  also  note  the  mix  of  commodity  type  (force  application  versus  logistics)  within  the  overall 
portfolio. 


77 


78  Initial  Examination  of  Defined  Metrics:  Helicopter  and  Satellite  Portfolios 


our  calculated  metrics  and  the  visualizations  chosen  to  package  this  information  for  a 
potential  policymaker. 

Portfolio  Composition  and  Latest  Milestone  Achieved 

To  review,  Tables  9.1  and  9.2  show  the  composition,  by  program  and  most  recent 
milestone  achieved,  of  the  helicopter  and  satellite  portfolio  in  each  year  of  our  analysis. 


Table  9.1 

Helicopter  Portfolio  Composition  Between  2002  and  2012 


Program 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

2010 

2011 

2012 

Army 

ARH 

B 

B 

B 

B 

CH-47F 

B 

B 

C 

C 

C 

C 

C 

C 

C 

c 

C 

Comanche 

B 

B 

UH-60M 

B 

B 

B 

C 

C 

C 

C 

C 

C 

c 

C 

AH-64D  Longbow  Apache3 

C 

C 

C 

C 

C 

C 

c 

C 

C 

LUH 

C 

c 

c 

C 

C 

c 

C 

AH-64E  Remanufacture3 

B 

B 

B 

B 

C 

c 

C 

AH-64E  New  Build3 

C 

c 

C 

CH-53K 

B 

B 

B 

B 

B 

B 

B 

B 

Navy 

H-1  Upgrades 

B 

B 

B 

B 

B 

B 

C 

C 

C 

C 

C 

MH-60R 

B 

B 

B 

B 

C 

C 

C 

C 

C 

C 

C 

MH-60S 

C 

C 

C 

C 

C 

C 

C 

C 

c 

c 

c 

VH-71 

B 

B 

B 

B 

NOTES:  A  red  B  indicates  that  the  program  has  passed  Milestone  B,  and  a  blue  C  indicates  that  the 
program  has  passed  Milestone  C.  A  blank  cell  in  the  table  suggests  that  the  program  has  not  started 
reporting  SAR  data,  has  been  cancelled,  or  is  no  longer  required  to  report  SAR  data.  Our  team  chose 
these  three  portfolios  both  to  test  our  methodology  and  present  examples  for  future  analysts  to  follow 
in  their  own  portfolio  analyses.  Again,  one  may  choose  from  a  variety  of  acquisition  characteristics  to 
create  a  portfolio  or  a  set  of  analogous  portfolios  for  comparison. 

3  Three  separate  Apache  acquisition  programs  are  covered  in  this  analysis.  The  AH-64D  Longbow 
Apache  was  reported  in  the  SARs  from  1997  through  2010.  It  is  referred  to  as  "Longbow  Apache"  in 
subsequent  figures  in  this  analysis.  The  AH-64E  Remanufacture  started  its  SAR  reporting  in  September 
2006  under  the  name  of  "Longbow  Apache-Block  III  (AB3)."  It  then  was  reported  under  the  following 
names:  Apache  Block  III  (AB3),  Apache  Block  IIIA  (AB3A  Remanufacture),  and  finally  as  AH-64E  Apache 
Remanufacture  (AH-64E  Remanufacture)  in  December  2012.  Even  with  the  multiple  name  changes,  it 
remained  the  same  acquisition  program.  The  AH-64E  New  Build  started  in  2010  and  was  formed  based 
on  the  2009  Nunn-McCurdy  outcome  of  the  AH-64E  Remanufacture  program.  The  first  name  for  the 
New  Build  program  was  Apache  Block  NIB  New  Build  (AB3B  New  Build)  in  2010  and  was  changed  to 
AH-64E  Apache  New  Build  (AH-64E  New  Build)  in  2012. 
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Table  9.2 

Satellite  Portfolio  Composition  Between  2002  and  2012 


Program 

2002 

2003  2004 

2005  2006 

2007 

2008 

2009  2010 

2011 

2012 

Air  Force 

AEHF 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

GBS 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

GPS  III 

B 

B 

B 

C 

C 

GPS  OCX 

B 

NAVSTAR  GPS 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

NPOESS 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

SBIRS-High 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

SBSS 

B 

B 

B 

B 

TSAT 

B 

WGS 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

Navy 

MUOS 

B 

B 

C 

C 

C 

C 

C 

C 

C 

NESP 

C 

C 

C 

NOTES:  A  red  B  indicates  that  the  program  has  passed  Milestone  B,  and  a  blue  C  indicates  that  the 
program  has  passed  Milestone  C.  A  blank  cell  in  the  table  suggests  that  the  program  has  not  started 
reporting  SAR  data,  has  been  cancelled,  or  is  no  longer  required  to  report  SAR  data. 

When  outlining  these  results,  a  key  observation  is  how  the  composition  of  both  portfo¬ 
lios  changes  over  time,  with  programs  coming  in  and  dropping  out  because  of  program 
completion  or  termination.  This  is  especially  important  in  the  satellite  portfolio,  which 
experiences  somewhat  more  flux  in  composition  over  the  period  under  consideration.  A 
second  observation  has  to  do  with  the  maturity  of  the  portfolios.  The  progression  of  Bs 
and  Cs  across  the  two  tables  suggests  two  very  different  trends.  Whereas  the  number 
of  Cs  included  in  the  helicopter  portfolio  increases  over  time,  the  number  of  Cs  in  the 
satellite  portfolio  remains  largely  the  same.  The  trend  in  the  helicopter  portfolio  clearly 
suggests  the  maturation  of  the  portfolio.  However,  using  the  milestone  metric  as  an 
indicator  of  maturity  in  the  satellite  program  does  not  show  this  same  trend.  This  may 
be  partially  because  many  satellite  programs  often  never  pass  Milestone  C  due  to  the 
nature  of  satellite  programs  themselves.2  Specifically,  for  satellite  programs,  the  first 


2  See  Department  of  Defense  Instruction  5000.02,  January  7,  2015:  “Operation  of  the  Defense  Acquisition 
System.”  Some  programs,  notably  spacecraft  and  ships,  will  not  produce  prototypes  during  EMD  for  use  solely 
as  test  articles  because  of  the  very  high  cost  of  each  article.  In  this  case,  the  first  articles  produced  will  be  tested 
and  then  fielded  as  operational  assets.  These  programs  may  be  tailored  by  measures  such  as  combining  the  devel- 
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development  unit  is  often  considered  the  first  operational  unit  as  well.  Because  of  the 
long  build  and  test  cycle  for  each  satellite,  cost  and  production  rates  do  not  ramp  up  as 
they  do  for  other  types  of  programs  (LRIP  and  FRP  have  no  meaning).3 

Figures  9.1  and  9.2  show  the  importance  of  including  key  contextual  informa¬ 
tion  within  a  metric  visualization.  Figure  9.1  shows  the  same  gradual  maturation 
of  programs  within  the  portfolio  and  the  shift  from  having  a  mix  of  programs  at 
Milestones  B  and  C  toward  a  portfolio  in  which  almost  all  programs  have  reached 
Milestone  C.  In  Figure  9.2,  we  do  not  see  a  significant  trend  in  terms  of  the  distribu¬ 
tion  of  programs  by  milestone  passed  for  the  satellite  portfolio.  The  number  of  pro¬ 
grams  at  Milestone  C  remains  largely  the  same,  although  the  programs  represented  do 
change  over  time.  One  reason  for  the  lack  of  variation  on  this  metric  may  be  due  to 
changes  in  portfolio  composition.  As  new  programs  enter  the  portfolio,  they  have  not 

Figure  9.1 

Helicopter  Programs,  by  Milestone,  2002  Through  2012 


RAND  MMG1 17118-9.1 


opment  and  initial  production  investment  commitments.  When  this  is  the  case,  a  combined  Milestone  B  and  C 
will  be  conducted.  Additional  decision  points  with  appropriate  criteria  may  also  be  established  for  subsequent  low 
rate  production  commitments  that  occur  prior  to  operational  test  and  evaluation  (OT&E)  and  a  Full  Rate  Pro¬ 
duction  Decision.  For  additional  policy  documentation  on  MDAs,  see  Under  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics),  memorandum  for  the  Secretary  of  the  Air  Force,  Subj:  Redelegation  of  Milestone 
Decision  Authority  (MDA),  January  4,  2006;  Acting  Under  Secretary  of  Defense  (Acquisition,  Technology  and 
Logistics),  memorandum  for  the  [Acting]  Secretary  of  Defense,  Subj:  Air  Force  Space  Programs — Change  in 
Milestone  Decision  Authority,  March  25,  2005. 

3  Because  of  data  availability,  the  numbers  differ  for  certain  metrics.  The  number  of  programs  included  in  each 
metric  calculation  is  included  in  individual  tables  with  each  visualization. 
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Figure  9.2 

Satellite  Programs,  by  Milestone,  2002  Through  2012 
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yet  reached  the  appropriate  milestone.  Another  may  be  the  nature  of  satellite  programs 
themselves  and  their  different  development  trajectory,  noted  above. 

Understanding  the  commodity  types  in  which  the  programs  in  a  given  acquisi¬ 
tion  portfolio  fall  may  also  be  important  because  it  provides  insight  into  the  mission 
sets  into  which  these  acquisition  programs  fall.  Figure  9.3  shows  that  the  helicopter 
portfolio  falls  into  two  DAMIR  categories,  force  application  and  logistics.  Over  time, 
the  relative  number  of  programs  in  the  logistic  category  appears  to  increase  as  com¬ 
pared  to  the  number  of  force  application  programs.  Satellite  programs  also  fall  into  two 
major  DAMIR  categories:  battlespace  awareness  and  net  centric  programs,  as  shown 
in  Figure  9.4.  Further  consideration  of  how  the  distribution  of  commodity  types  in  a 
given  portfolio  influences  portfolio  status  and  performance  could  be  a  valuable  line  of 
investigation  during  a  complete  portfolio  analysis. 

Program  Size 

The  total  program  value  metric  is  important  because  it  gives  analysts  a  sense  of  the 
average  dollar  value  of  programs  in  the  portfolio  as  well  as  the  range  of  program  sizes 
within  the  portfolio  (with  program  size  measured  using  total  cost  in  dollars).  Our 
visualizations  also  provide  insight  into  how  the  portfolio  changes  on  this  dimension 
over  the  assessment  period.  Figure  9.5  for  the  helicopter  portfolio  shows  almost  no 
change  in  total  program  value  when  considering  the  portfolio  as  a  whole.  Although  the 
median  does  appear  to  move  up  slightly,  this  change  is  statistically  significant  only  in 
the  period  2008  to  2010.  Another  notable  trend  is  the  increase  in  the  range  of  program 
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Figure  9.3 

Helicopter  Programs,  by  DAMIR  Portfolio,  2002  Through  2012 
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Figure  9.4 

Satellite  Programs,  by  DAMIR  Portfolio,  2002  Through  2012 
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Figure  9.5 

Total  Average  Program  Value,  Helicopter  Portfolio,  2002  Through  2012 
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values  in  2006  through  2008.  One  can  observe  this  increase  by  looking  at  the  spread 
of  the  whiskers  as  well  as  the  length  of  the  interquartile  range.  Turning  to  the  satellite 
portfolio,  Figure  9.6,  we  see  much  less  variation  over  time.  There  is  a  small,  statistically 
significant  increase  in  the  median  and  an  apparent  increase  in  the  range  or  spread  of 
total  cost  values,  but  one  that  is  much  less  dramatic  than  that  observed  for  the  heli¬ 
copter  portfolio.  This  greater  stability  in  total  program  value  in  the  satellite  portfolio  is 
somewhat  surprising  given  the  greater  fluctuation  in  the  portfolio’s  composition  over 
time.  It  appears  that  even  as  the  composition  of  the  portfolio  changes,  the  size  of  the 
programs  included  does  not  change  substantially  over  time. 

Percentage  of  Funds  Remaining 

Looking  at  the  percentage  of  procurement  and  RDT&E  funds  remaining  can  be 
another  way  of  thinking  about  program  maturity,  as  a  more  mature  program  will  have 
fewer  funds  remaining.  Figure  9.7A  and  9.7B  show  funds  remaining  for  the  helicopter 
portfolio.  Looking  first  at  procurement  funds,  two  trends  are  worth  discussing.  First, 
there  is  a  clear  decline  in  the  median  over  the  period  under  consideration.  This  change 
is  statistically  significant  in  all  years,  and  the  results  suggest  a  decrease  of  about  5  per¬ 
cent  in  funds  remaining  on  average  over  the  period.  The  second  trend  worth  noting  is 
the  large  increase  in  range,  which  occurs  because  although  some  programs  do  gradu¬ 
ally  spend  their  procurement  budgets,  other  helicopter  programs  still  have  100  per¬ 
cent  of  their  procurement  budgets  remaining  even  in  the  final  year  of  our  analysis. 
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Figure  9.6 

Total  Average  Program  Value,  Satellite  Portfolio,  2002  Through  2012 


•  Change  from  2002  to 
2012  is  statistically 
significant  at  p  <  0.03 
(and  positive) 

•  Change  from  2007  to 
2012  is  not  statistically 
significant  at  p  =  0.32 

•  Substantive 
significance:  Small 


2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012 

Number  of 

programs:  7  7  6  9  10  10  10  8  8  7  7 


30 

'in 

c 

O 

1  25 


3  20 


15 


ns  10 


U) 

O 

&- 

Q. 

Id 

+■» 

o 


Extreme  values 
not  shown 


TT  T  TT 


X 


ITT 


I 


I 


RAND  MG1 171 18-9.6 


Figure  9.7A  shows  the  change  in  RDT&E  funds  remaining  over  the  same  time  period. 
Once  again,  we  see  a  gradual  decline  in  the  median  funds  remaining,  which  is  consis¬ 
tent  with  previous  graphs  that  suggested  a  maturing  helicopter  portfolio.  This  decrease 
is  once  again  statistically  significant  overall,  although  the  year-on-year  changes  are  not 
statistically  significant.  We  can  also  see  a  sizeable  decrease  in  the  range  until  2010,  as 
most  programs  appear  to  converge  toward  having  fully  spent  their  RDT&E  funds. 
Notably,  the  range  increases  again  in  2011,  reflecting  an  infusion  of  new  RDT&E 
funds  received  by  the  AH-64E  Remanufacture  in  2010  (AH-64E  is  an  outlier  in  2010 
as  a  result  and  is  not  shown  in  the  figure).  These  new  funds  were  intended  to  support 
additional  improvement  in  the  AH-64E’s  combat  capability. 

Figure  9.7C  shows  the  percentage  of  RDT&E  funds  remaining  for  the  satellite 
portfolio.  The  figure  shows  that  over  the  period  under  consideration  in  this  assessment, 
programs  in  the  satellite  portfolio  have  gradually  spent  their  RDT&E  funding.  Little 
variation  appears  in  the  amount  of  this  type  of  funding  left  by  2012.  As  noted  else¬ 
where  in  the  report,  this  is  evidence  that  the  portfolio  is  becoming  more  mature,  with 
more  programs  entering  the  procurement  phase.  Figure  9.7D  shows  the  percentage  of 
procurement  funds  remaining  for  the  satellite  portfolio.  We  can  see  that  although  the 
statistical  analysis  suggests  a  decline  in  procurement  dollars  remaining  on  average  over 
the  period  under  consideration,  the  figure  suggests  a  very  different  trend  than  that 
observed  for  the  helicopter  portfolio.  For  the  satellite  portfolio,  the  median  seems  to 
jump  around  a  bit  more,  and  the  range  spans  from  0  to  100  percent  funds  remaining 
in  all  years.  This  partly  reflects  the  greater  flux  (with  programs  more  frequently  enter- 


Figure  9.7 

Percentage  of  Funds  Remaining,  2002  Through  2012 
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ing  and  leaving  the  portfolio)  observed  in  the  satellite  portfolio’s  composition:  As  new 
programs  enter,  they  may  not  have  spent  procurement  dollars  yet.  However,  the  differ¬ 
ent  trend  is  also  confirmatory  evidence  that  the  satellite  and  helicopter  portfolios  have 
some  significant  differences  in  overall  status  and  likely  risk  as  well. 

Portfolio  Churn:  Quantity  Changes 

When  assessing  the  effects  of  quantity  change  on  portfolio  status,  we  focused  on 
churn,  or  the  percentage  of  programs  experiencing  any  quantity  change  in  each  year  of 
our  analysis.  As  noted  above,  changes  in  quantity  may  not  indicate  serious  problems. 
Instead,  a  change  in  quantity  (a  decrease  in  particular)  can  be  driven  by  a  change  in 
need  for  a  weapons  system.  It  could  also  be  due  to  affordability  within  a  given  port¬ 
folio.  Although  not  always  indicative  of  a  problem  within  the  affected  program,  high 
levels  of  churn  are  sometimes  associated  with  unit  cost  growth,  schedule  slippage,  and 
other  performance  problems  as  they  may  significantly  alter  manufacturing  schedules, 
leading  to  a  less-than-optimal  economic  rate  of  production.4  Looking  at  our  churn 
visualizations  in  Figures  9.8  and  9.9,  we  can  see  that  both  the  satellite  and  helicopter 
portfolios  experienced  varying  amounts  of  churn  in  the  years  of  our  assessment.  For 
the  helicopter  portfolio,  quantity  churn  primarily  occurs  because  of  increases  in  quan¬ 
tity,  with  the  highest  amounts  in  the  portfolio  occurring  between  2003-2004,  2005- 
2006,  and  2008-2009.  However,  analysis  of  the  year-on-year  differences  suggests  that 
none  of  these  changes  are  statistically  significant.  The  same  is  true  when  we  compare 
the  churn  in  2002  to  that  in  2007  and  that  in  2012.  The  lack  of  statistical  significance 
does  not  mean  that  changes  in  quantity  do  not  occur  and  are  not  important;  it  means, 
rather,  that  any  observed  changes  in  quantity  are  more  or  less  random  over  time.  It  is 
worth  noting  that  a  good  amount  of  the  growth  in  quantity  in  the  early  years  of  our 
time  period  is  due  to  high  demand  for  helicopters  within  the  Marine  Corps  and  the 
Army  because  of  operations  in  Iraq  and  Afghanistan.  The  decrease  seen  toward  the 
end  of  the  period  may  similarly  reflect  a  drop  off  in  demand  as  these  operations  draw 
to  a  close. 

There  seems  to  be  even  more  churn  for  the  satellite  portfolio,  on  both  the  posi¬ 
tive  and  negative  sides,  especially  between  2006  and  2009.  In  this  case,  some  of  the 
year-on-year  changes  are  statistically  significant.  Specifically  the  churn  values  mea¬ 
sured  between  2006-2007,  2007-2008,  2008-2009,  and  2009-2010  are  statistically 
significant  compared  to  the  period  prior.  In  the  2006-2007  period,  there  are  a  greater 
number  of  quantity  changes  and  specifically  a  greater  number  of  positive  quantity 
changes.  In  the  2007-2008  period,  the  trend  is  in  the  opposite  direction,  with  a  larger 
number  of  negative  quantity  changes.  In  the  2008-2009  period,  there  are  several  large 
positive  quantity  changes.  Finally,  in  the  2009-2010  period,  there  is  little  change  in 


4  For  a  further  examination  of  this  concept,  see  Congressional  Budget  Office,  “Effects  of  Weapons  Procurement 
Stretch-Outs  on  Costs  and  Schedules,”  November  1987. 
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Figure  9.8 

Percentage  Change  in  Current  Estimate  Quantity,  Year  by  Year,  Helicopter  Portfolio,  2002 
Through  2012 
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Figure  9.9 

Percentage  Change  in  Current  Estimate  Quantity,  Year  by  Year,  Satellite  Portfolio,  2002 
Through  2012 
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either  direction.  Looking  at  five-year  changes  and  the  satellite  portfolio,  the  change 
between  2002  and  2007  is  statistically  significant,  but  the  change  between  2002  and 
2012  is  not.  This  reflects  the  fact  that  there  was  significantly  more  churn  in  the  middle 
years  of  our  assessment  than  at  the  start  or  end  of  the  period  we  consider  in  our  analysis. 

When  studying  portfolio  churn,  it  is  important  to  note  that  what  constitutes  a 
“high  level”  of  churn  can  also  vary  across  portfolio  types  because  of  program  charac¬ 
teristics  or  simply  the  quantity  levels  involved.  For  example,  both  the  ARH  between 
2005  and  2006  and  WGS  between  2006  and  2007  experienced  a  roughly  40  percent 
increase  in  quantity.  However,  ARH  increased  from  368  units  to  512,  whereas  the 
WGS  program  increased  from  five  to  seven  satellites.  Satellite  programs,  with  their  typ¬ 
ically  low  quantity  levels,  are  more  sensitive  to  quantity  changes  than  are  the  higher- 
production  helicopter  programs.  It  is  also  worth  noting  that  at  least  some  of  this  churn 
may  be  driven  by  changes  in  the  composition  of  the  satellite  and  helicopter  portfolio. 
Additional  analysis  in  an  assessment  phase  might  explore  what  drove  these  changes 
and  might  attempt  to  make  some  diagnosis  about  what  drove  these  changes  in  quan¬ 
tity  and  whether  they  signal  performance  problems  or  respond  to  some  other  type  of 
external  trigger  or  event. 


Performance  Metrics 

With  the  descriptive  metrics  providing  context,  we  then  turned  to  the  metrics  that 
describe  the  cost  and  schedule  performance  outcomes  of  the  portfolios.  These  perfor¬ 
mance  metrics  include  the  Nunn-McCurdy  breaches  representing  unit  cost  growth 
and  the  APB  breaches  representing  deviation  from  schedule,  performance,  and  cost 
from  the  established  APB.  As  stated  above,  although  we  wished  to  assess  the  amount 
of  KPPs  at  or  above  threshold,  data  on  KPPs  were  not  consistently  reported  (in  SARs 
and  DAES)  across  services  and  programs. 

Nunn-McCurdy  and  APB  Breaches 

Tables  9.3  through  9.5  show  the  number  and  types  of  Nunn-McCurdy  and  APB 
breaches  within  each  portfolio  over  the  period  under  consideration.5  We  have  used 
tables  to  display  this  information  because  this  seems  to  be  the  clearest  way  to  represent 
portfolio  performance  using  these  metrics.  In  the  tables,  the  numbers  represent  total 
breaches  of  each  type  within  the  portfolio  in  a  given  year.  This  means  that  if  a  program 
has  three  different  types  of  breaches  in  one  year,  all  three  breaches  will  be  recorded  in 
the  table.  Table  9.3  provides  insight  into  the  difference  between  the  satellite  and  heli¬ 
copter  portfolios  in  terms  of  Nunn-McCurdy  breaches.  The  figure  suggests  somewhat 


5  Policy  changes  during  our  portfolio  time  frame  changed  the  way  that  Nunn-McCurdy  breaches  were  reported. 
The  discussion  on  this  issue  can  be  found  in  Chapter  Eleven. 
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Table  9.3 

Significant  and  Critical  Nunn-McCurdy  Breaches  Experienced,  Helicopter  and  Satellite 
Portfolios,  2002  Through  2012 


Year 

Current 

Original 

APUC, 

critical 

APUC, 

significant 

PAUC, 

critical 

PAUC, 

significant 

APUC, 

critical 

APUC, 

significant 

PAUC, 

critical 

PAUC, 

significant 

2002 

1 

1 

2003 

2004 

2 

1 

2005 

1 

1 

1 

1 

1 

1 

2006 

1 

1 

1 

1 

1 

1 

2007 

1 

1  1 

1 

1 

1 

1 

1 

1 

2008 

2 

1 

1 

1 

2 

1 

1 

1 

2009 

2  1 

1 

1 

1 

1  2 

1 

2010 

1 

1 

1 

2011 

1 

1 

2012 

1 

1 

Helicopter  portfolio 


Satellite  portfolio 


more  breaches  in  the  satellite  portfolio,  although  we  did  not  consider  the  statistical 
significance  of  this  difference. 

Looking  at  Tables  9.4  and  9.5,  we  can  consider  APB  breaches  by  number  and 
type  across  the  years  of  our  analysis  as  they  are  reported  in  the  SARs  in  the  Threshold 
Breaches  section.  In  general,  an  APB  breach  occurs  when  the  program  exceeds  the 
targets  for  cost,  schedule,  and  performance  defined  in  the  APB.  There  are  a  number  of 
different  types  of  APB  breaches  each  with  its  own  definition  or  criteria.  We  focus  here 
on  nine  types  of  APB  breaches: 

•  schedule:  objective  schedule  value  plus  six  months. 

•  performance:  failure  to  achieve  standards  defined  in  APB  document 

•  RDT&E  cost:  objective  cost  value  plus  10  percent 

•  procurement  cost:  objective  cost  value  plus  10  percent 

•  MILCON  cost:  objective  cost  value  plus  10  percent 

•  acquisition  operation  and  maintenance  (O&M)  cost:  objective  cost  value  plus 
10  percent 

•  O&S  cost:  objective  cost  value  plus  10  percent 

•  APUC:  breach  occurs  when  cumulative  program  cost  increases  >  15  percent  of 
the  approved  current  baseline  or  30  percent  above  the  original  baseline 


Table  9.4 

APB  Breaches,  by  Program  and  Year,  Helicopter  Portfolio 


Breach 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

2010 

2011 

2012 

Schedule 

CH-47F 

MH-60R 

H-1  Upgrades 
MH-60S 

H-1  Upgrades 

ARH 

H-1  Upgrades 
MH-60S 

VH-71 

ARH 

MH-60S 

VH-71 

ARH 

MH-60S 

VH-71 

CH-53K 

MH-60S 

UH-60M 

CH-53K 

UH-60M 

CH-53K 

MH-60S 

MH-60S 
AH-64E 
(New  Build) 

Performance 

UH-60M 

UH-60M 

UH-60M 

MH-60R 

UH-60M 

MH-60R 

MH-60S 

H-1  Upgrades 
MH-60R 

MH-60S 

Longbow 

MH-60R 

MH-60S 

RDT&E  cost 

MH-60R 

H-1  Upgrades 
MH-60S 

ARH 

ARH 

CH-53K 

MH-60S 

CH-53K 

CH-53K 

Procurement 

MH-60R 

H-1  Upgrades 

Longbow 

MH-60S 

ARH 

Longbow 

AH-64E 

Remanufacture 

ARH 

Longbow 

ARH 

Longbow 

CH-53K 

Longbow 

MH-60R 

AH-64E 

Remanufacture 

CH-53K 

Longbow 

UH-60M 

CH-53K 

MILCON  cost 

O&S  cost 

MH-60S 

MH-60S 

Longbow 

MH-60S 

UH-60M 

MH-60S 

AH-64E 

Remanufacture 

MH-60S 

UH-60M 

ARH 

UH-60M 

LUH 

MH-60R 

MH-60S 

UH-60M 

CH-53K 

LUH 

MH-60S 

UH-60M 

CH-53K 

LUH 

MH-60S 

UH-60M 

LUH 

MH-60S 

UH-60M 

O&M  cost 

PAUC 

MH-60R 

H-1  Upgrades 

ARH 

AH-64E 

Remanufacture 

APUC 

MH-60R 

H-1  Upgrades 

ARH 

AH-64E 

Remanufacture 

Longbow 

Longbow 

NOTE:  A  change  in  the  number  and  types  of  breaches  between  2007  and  2012  is  statistically  significant,  but  the  difference  between  2002  and  2012 
is  not. 
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Table  9.5 

APB  Breaches,  by  Program  and  Year  Experienced,  Satellite  Portfolio 


Breach 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

2010 

2011 

2012 

Schedule 

NAV  GPS 

NAV  GPS 

NAV  GPS 

GBS 

NPOESS 

GBS 

AEHF 

AEHF 

GBS 

GBS 

AEHF 

SBIRS 

SBIRS 

NAV  GPS 

NPOESS 

GBS 

GBS 

MUOS 

MUOS 

GBS 

NPOESS 

SBIRS 

NAV  GPS 

GPS  III 

NAV  GPS 

NAV  GPS 

NAV  GPS 

WGS 

NPOESS 

MUOS 

NPOESS 

NPOESS 

WGS 

SBIRS 

NAV  GPS 

SBIRS 

NPOESS 

SBIRS 

SBSS 

Performance 

AEHF 

AEHF 

NAV  GPS 

NPOESS 

GBS 

AEHF 

NAV.GPS 

NAV_GPS 

NPOESS 

RDT&E  cost 

NPOESS 

GBS 

NAV  GPS 

NPOESS 

NPOESS 

AEHF 

AEHF 

MUOS 

MUOS 

NAV  GPS 

NPOESS 

NPOESS 

MUOS 

SBIRS 

SBIRS 

SBIRS 

SBSS 

Procurement 

SBIRS 

NPOESS 

NPOESS 

AEHF 

AEHF 

AEHF 

GBS 

AEHF 

GBS 

cost 

WGS 

NPOESS 

NPOESS 

GBS 

SBIRS 

GBS 

SBIRS 

SBIRS 

WGS 

SBIRS 

SBIRS 

MILCON  cost 

O&S  cost 

NPOESS 

SBIRS 

SBIRS 

SBIRS 

SBIRS 

SBIRS 

WGS 

SBIRS 

AEHF 

WGS 

O&M  cost 

PAUC 

SBIRS 

NPOESS 

GBS 

NPOESS 

AEHF 

WGS 

NPOESS 

WGS 

NPOESS 

NPOESS 

SBSS 

APUC 

SBIRS 

NPOESS 

NPOESS 

AEHF 

AEHF 

AEHF 

WGS 

NPOESS 

NPOESS 

WGS 

NOTES:  A  change  in  the  number  and  types  of  breaches  between  2007  and  2012  is  not  statistically  significant.  A  change  between  2002  and  2012  is  also 
not  statistically  significant. 
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•  PAUC:  breach  occurs  when  cumulative  program  cost  increases  >  15  percent  of  the 

approved  current  baseline  or  30  percent  above  the  original  baseline.6 

We  can  compare  the  number  and  types  of  APB  breaches  not  only  across  portfo¬ 
lios  but  also  over  time  to  explore  whether  the  number  and  types  of  breaches  in  2002 
are  the  same  or  different  from  those  in  other  years,  such  as  2007  and  2012.  When  we 
conduct  this  analysis  for  the  helicopter  portfolio,  we  find  that  the  difference  is  statis¬ 
tically  significant  when  we  compare  the  number  of  breaches  in  2007  and  2012  (and 
suggests  an  improvement),  but  it  is  not  statistically  significant  when  we  compare  the 
number  of  breaches  in  2002  and  2012.  Neither  the  2002  to  2012  nor  the  2007  to  2012 
difference  is  statistically  significant  for  the  satellite  portfolio,  however.  Tables  9.4  and 
9.5  also  show  the  specific  programs  experiencing  a  breach  in  each  year  of  the  analysis. 
These  tables  are  useful  for  analysts  interested  in  knowing  not  only  how  many  total 
breaches  occurred  (across  all  programs)  but  also  which  specific  programs  appear  to 
drive  these  breaches. 

Unit  Cost  Growth 

For  unit  cost  growth — a  metric  of  high  importance  to  policymakers  and  analysts — we 
considered  a  number  of  different  visualizations.  First,  we  looked  at  box-and-whisker 
plots,  the  interpretation  of  which  was  addressed  in  Chapter  Eight.  It  is  worth  noting 
that  for  these  graphs,  although  all  programs  were  included  in  the  data  analysis,  we 
have  excluded  outlier  data  points  from  the  graph  (the  list  of  programs  with  data  points 
not  included  is  at  the  bottom  of  each  figure).  These  outliers  were  included  in  the  graph 
for  presentational  purposes  only.  Outliers  are  excluded  because  they  lie  so  far  outside 
the  box-and-whisker  part  of  the  graph  that  they  distort  the  graph,  making  it  difficult 
to  observe  year-on-year  changes.  If  desired,  an  analyst  could  make  a  different  presen¬ 
tational  decision.  Figures  9.10A  and  9.10B  show  these  plots  for  the  PAUC  and  APUC 
for  the  helicopter  portfolio  using  the  current  baseline  (and  without  the  extreme  values 
shown).  Several  trends  are  worth  noting.  One  is  the  general  decline  in  the  median  for 
both  the  PAUC  and  the  APUC,  particularly  in  the  second  half  of  the  period  under 
review,  after  about  2007.  However,  it  is  important  to  note  that  this  trend  reaches 
statistical  significance  for  both  the  PAUC  and  APUC  only  when  the  years  2007,  2008, 
2009,  and  2010  are  compared  with  2012.  Also,  percentage  unit  cost  growth  in  2008 
and  2009  was  statistically  different  and  higher  than  cost  growth  in  other  years.  It  is 
these  statistically  significant  changes  that  we  should  be  most  interested  in  when  con¬ 
ducting  the  portfolio  assessment  because  they  indicate  a  significant  trend  shift,  pos¬ 
sibly  indicative  of  a  change  in  overall  performance.  Observing  increases  and  decreases 
in  the  median  percentage  unit  cost  growth  is  also  valuable,  but  from  the  perspective  of 
any  acquisition  executive,  longer-term  and  meaningful  trends  are  the  more  important 


6 


Department  of  Defense  Instruction  5000.02,  2015,  pp.  59-61. 


Figure  9.10 

Percentage  Unit  Cost  Growth,  APUC  and  PAUC  Current  Baseline,  2002  Through  2012 


A.  Helicopter,  APUC  current  baseline 
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measures  of  performance.  Even  then,  the  substantive  size  of  the  observed  effect  size  is 
only  moderate.  We  can  also  consider  the  change  in  the  range  or  the  spread  of  the  data. 
After  increasing  somewhat  between  2002  and  2008,  the  range  also  appears  to  shrink, 
likely  because  several  programs  with  higher  cost  growth  are  terminated  or  rebaselined. 
To  understand  the  factors  driving  these  trends,  we  would  need  to  separately  consider 
the  unit  cost  growth  in  individual  programs  within  the  portfolio.  We  look  into  some 
of  these  trends  in  Chapter  Five,  although  a  comprehensive  assessment  is  still  outside 
the  scope  of  this  report. 

For  the  satellite  portfolio,  we  use  the  same  unit  cost  visualizations  to  present  our 
metrics  of  portfolio  status.  Figure  9.10C  and  9.10D  show  the  box-and-whisker  plots 
for  the  satellite  portfolio  focused  on  the  APUC  and  PAUC  metrics.  First,  it  is  worth 
noting  that  none  of  the  year-to-year  changes  in  the  PAUC  and  APUC  of  the  portfolio 
are  statistically  significant.  This  means  that  overall,  portfolio  status  on  unit  cost  does 
not  seem  to  change  all  that  much.  The  PAUC  graph  illustrates  this  lack  of  change. 
The  median  appears  to  hover  around  zero  and  although  the  range  of  the  data  does 
vary  somewhat  over  time,  there  does  not  appear  to  be  a  consistent  pattern.  Ffowever, 
the  APUC  graph  does  show  some  more  significant  changes.  The  median  once  again 
appears  to  be  fairly  steady,  at  or  around  zero.  Ffowever,  there  is  more  fluctuation  in 
the  range,  which  goes  from  very  narrow  in  2002  to  very  wide  in  2008  and  2009.  This 
change  appears  to  be  driven  by  a  few  poor  performers  in  2008  and  2009,  namely,  the 
AEFfF,  NPOESS,  and  WGS  programs,  which  have  high  unit  cost  growth  in  these 
years.  It  is  important  to  note  that  both  the  AEFfF  and  NPOESS  were  later  cancelled 
and  so  drop  out  of  our  portfolio.  Partly  for  this  reason,  the  cost-growth  range  returns 
to  its  original  spread  by  the  end  of  the  assessment  period  in  2013.  In  this  case,  the  can¬ 
cellation  of  the  programs  improves  the  overall  performance  of  the  portfolio. 

Figures  9.11  and  9.12  show  a  different  way  to  assess  the  unit  cost  growth  perfor¬ 
mance  at  the  portfolio  level.  Although  the  bars  for  each  program  provide  insight  into 
the  performance  of  specific  programs  in  each  of  the  years  shown,  the  overall  graph 
shows  a  snapshot  of  the  unit  cost  growth  distribution  across  all  programs  in  the  port¬ 
folio  at  a  single  point  in  time.  Figures  9.11  and  9.12  allow  us  to  look  at  the  distribution 
of  cost  growth  by  program.  We  focus  here  on  the  changes  between  2002  and  2012 
(Figure  9.11A)  and  2007  and  2012  (Figure  9.1  IB),  although  similar  comparisons  could 
be  completed  for  all  years.  These  graphs  can  be  used  to  identify  some  of  the  extreme 
values  (on  the  positive  and  negative  sides)  and  to  see  how  programs  perform  relative  to 
others  within  the  portfolio.  For  example,  Figure  9.11A  highlights  the  high  cost  growth 
experienced  by  the  RAFi-66  Comanche  in  2002  and  the  negative  cost  growth  experi¬ 
enced  by  the  UFf-60  Black  Ffawk  program  in  2012.  Figure  9.11B  similarly  highlights 
the  apparently  strong  unit  cost  performance  of  the  VH-71  Presidential  Ffelicopter  and 
the  sizeable  cost  growth  experienced  by  the  Armed  Reconnaissance  Helicopter  (ARH) 
in  2007.  However,  when  considering  the  2012  performance  of  the  VH-71,  it  is  impor¬ 
tant  to  note  that  2012  SAR  was  later  judged  to  be  inaccurate.  In  reality,  the  program 


Figure  9.11 

Percentage  Unit  Cost  Growth  APUC  and  PAUC  Current  Baseline,  Helicopter  Portfolio,  2002,  2007,  and  2012 
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Figure  9.12 

Percent  Unit  Cost  Growth,  APUC  and  PAUC  Current  Baseline,  Satellite  Portfolio,  2002,  2007,  and  2012 
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was  performing  poorly  and  had  serious  problems  that  were  at  odds  with  the  trends  sug¬ 
gested  by  the  graphs  and  metrics  in  this  report  and  the  data  that  underlie  them.  This 
suggests  that  the  data  being  reported  by  program  directors  warrants  additional  scru¬ 
tiny  and  highlights  an  important  point:  Our  ability  to  analyze  and  assess  program  and 
portfolio  performance  is  only  as  good  as  the  data  provided.  If  data  are  false  or  inaccu¬ 
rate,  then  the  metrics  and  visualizations  produced  based  on  these  data  will  be  similarly 
skewed.  Analysts  conducting  portfolio  assessments  should,  in  addition  to  calculating 
metrics,  investigate  the  programs  being  studied  in  additional  detail  to  understand  pro¬ 
gram  history  and  other  similar  factors. 

A  final  notable  change  visible  in  the  2012  graph  is  the  negative  cost  growth  expe¬ 
rienced  by  the  AH-64E  New  Build  in  2012.  This  program  was  not  active  in  either 
2002  or  2007.  To  complete  a  comprehensive  analysis  of  unit  cost  growth,  an  analyst 
would  need  to  consider  similar  charts  for  all  years.  Looking  only  at  2002,  2007,  and 
2012  disguises  programs  with  high  unit  cost  growth  and  PAUC  or  APUC  breaches 
in  other  years.  For  example,  the  AH-64E  Remanufacture  has  an  APUC  and  PAUC 
breach  in  2009  that  is  missed  by  considering  only  the  three  years  shown  in  Figure  9.11. 

Several  programs  stand  out  for  the  satellite  portfolio  (Figures  9.12A  and  9.12B). 
In  2002,  the  NPOESS  and  CBS  programs  show  sizeable  amounts  of  unit  cost  growth, 
but  the  NESP  program  has  a  negative  change  in  unit  cost.  In  2012,  the  CBS  program 
has  a  significant  negative  change  in  unit  cost,  suggesting  that  the  program  may  have 
been  able  to  turn  things  around.  Also  showing  negative  unit  cost  growth  are  the  AEHF 
and  SBIRS-High  programs.  In  2007,  all  trends  are  dwarfed  by  the  large  increase  in 
unit  cost  experienced  by  the  NPOESS  program. 

The  final  unit  cost  visualization  that  we  consider  in  our  analysis  is  the  heat  map 
(Figure  9.13).  The  different  colors  in  the  heat  map  represent  program  performance  on 
unit  cost  in  each  year  of  the  analysis.  As  noted  above,  by  reading  down  a  column,  an 
analyst  can  get  a  sense  of  the  portfolio’s  unit  cost  performance  in  any  given  year.  Read¬ 
ing  across  any  given  row  offers  insight  into  the  performance  on  unit  cost  growth  of  any 
single  program  over  the  full  period  under  consideration.  Viewing  the  map  as  a  whole, 
then,  also  tells  the  analyst  about  trends  in  unit  cost  performance  over  time.  Looking  at 
this  trend  for  the  helicopter  portfolio  suggests  a  gradual  trend  toward  green  or  a  gradual 
improvement  of  the  portfolio  on  this  metric.  Elowever,  this  chart  does  not  consider  the 
statistical  significance  of  any  change.  The  heat  maps  can  also  help  to  identify  patterns 
in  the  data.  For  example,  for  both  the  ARH  and  the  Longbow  Apache,  we  observe  that 
APUC  cost  growth  is  at  its  highest  right  before  the  program  is  cancelled.  However,  the 
H-l  Upgrades  program  has  a  similar  pattern  between  2006  and  2008  but  manages  to 
improve  performance.  In  our  analysis  of  the  helicopter  portfolio  in  Chapter  Ten,  we 
investigate  the  program  differences  that  allowed  the  H-l  Upgrades  program  to  per¬ 
form  better  after  the  2006  to  2008  downturn  and  to  continue,  whereas  other  programs 
suffering  similar  downturns  were  later  cancelled.  The  AH-64E  Remanufacture  simi¬ 
larly  has  poor  unit  cost  growth  performance  in  its  early  years,  but  seems  to  improve 
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Figure  9.13 

"Heat  Map"  of  Unit  Cost  Growth  Performance,  Helicopter  Portfolio,  2002  Through  2012 
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markedly  starting  in  2010  following  a  2009  breach  in  PAUC  and  APUC.  Notably, 
however,  this  breach  was  due  to  an  Army  decision  on  accounting  and  cost-reporting 
rather  than  a  change  in  the  program’s  performance.7  The  improvement  in  unit  cost 
growth  followed  a  2010  rebaseline. 

Finally,  looking  at  the  heat  map  of  the  satellite  portfolio  (Figure  9.14),  we  once 
again  can  observe  changes  in  the  portfolio’s  performance  on  this  dimension  at  a  single 
moment  as  well  as  over  time  and  can  also  assess  changes  in  the  status  of  individual 
programs.  For  the  satellite  portfolio,  we  can  see  a  clear  trend  of  increasing  green  and 
decreasing  red,  suggesting  a  gradual  improvement  in  the  portfolio’s  APUC  perfor¬ 
mance.  Flowever,  there  appears  to  be  less  red  at  the  start  of  our  analysis  in  2002  than 
was  the  case  for  the  helicopter  portfolio.  At  the  same  time,  there  are  also  satellite  pro¬ 
grams  with  significant  periods  of  red,  suggesting  high  levels  of  unit  cost  growth  in 
these  years.  For  example,  looking  at  the  APUC  graph,  the  AEFiF  program  has  high 
unit  cost  for  a  significant  portion  of  the  assessment  period,  as  does  the  NPOESS  pro¬ 
gram  and  the  NAVSTAR  GPS  program.  Flowever,  although  the  NPOESS  program  is 
cancelled  by  2009,  the  AEFiF  program  appears  to  turn  its  performance  around  start¬ 
ing  in  2011.  Additional  analysis  in  an  assessment  phase  might  highlight  differences 
between  the  two  programs  that  can  explain  these  different  outcomes.  Looking  at  the 
PAUC,  we  again  see  a  trend  to  green,  as  well  as  more  limited  pockets  of  red,  or  poor 
cost-growth  performance.  The  NPOESS  and  SBSS  Block  20  program  appear  to  have 
the  most  substantial  and  sustained  unit  cost  growth.  Flowever,  there  are  several  pro¬ 
grams  with  short  periods  of  high  unit  cost  growth,  including  WGS,  MUOS,  AEFiF, 
and  CBS.  Each  of  these  programs  has  been  able  to  overcome  this  cost  growth  issue, 
however.  Additional  analysis  could  explore  how  and  why  some  programs  are  able  to 
overcome  cost  growth  problems,  but  others  are  not. 


Multimetric  Charts 

We  also  considered  two  multimetric  charts  (charts  that  consider  more  than  one  metric 
at  a  time)  for  each  portfolio.  First,  we  looked  at  bubble  charts,  which  consider  both 
unit  cost  and  program  size.  Second,  we  looked  at  radar  charts,  which  provide  insight 
into  the  aggregate  performance  of  a  program  across  several  different  metrics. 

Figure  9.15  shows  the  bubble  chart  for  the  helicopter  portfolio  and  compares 
across  three  years,  2002,  2007,  and  2012.  As  noted  in  Chapter  Eight,  an  analyst  would 
be  most  concerned  about  programs  in  the  upper  right  quadrant  of  the  graph,  particu¬ 
larly  when  these  programs  are  large  (represented  by  large  bubbles).  The  best  performers 
will  be  located  in  the  lower  left  quadrant  of  the  graph.  For  example,  in  2007,  the  ARFi 
program  warrants  some  concern  because  it  is  high  in  the  upper  right  quadrant  whereas 
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Figure  9.14 

"Heat  Map"  of  Unit  Cost  Growth  Performance,  Satellite  Portfolio,  2002  Through  2012 
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Figure  9.15 

Unit  Cost  Growth  and  Program  Size,  Helicopter  Portfolio,  2002,  2007,  and  2012 
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the  VH-71  appears  to  show  strong  performance  in  the  lower  left  (but  as  described 
above,  this  apparent  performance  is  based  on  poor  quality  data  and  is  misleading). 

Also  performing  well  in  2012  is  the  AH-64E  New  Build.  An  analyst  would  want 
to  pay  special  attention  to  the  UH-60M  Black  Hawk,  MH-60R,  and  CH-53K  as  large 
programs  whose  performance  might  be  more  meaningful  for  an  analyst  than  the  per¬ 
formance  of  an  average  program.  In  2012,  for  instance,  the  fact  that  the  Black  Hawk 
program  has  moved  into  the  lower  left  quadrant  is  a  positive  one  for  the  performance 
of  the  overall  portfolio. 

Figure  9.16  shows  the  same  type  of  plot  for  the  satellite  portfolio  in  the  same  three 
years,  2002,  2007,  and  2012.  Already  in  2002,  the  performance  of  the  NPOESS  pro¬ 
gram  appears  somewhat  problematic,  especially  given  the  fact  that  it  is  a  relatively  siz¬ 
able  program.  The  SBIRS-High  program  also  shows  some  cost  growth  as  does  the  GBS 
program,  although  this  program’s  small  size  decreases  the  level  of  concern  about  its 
effect  on  portfolio  performance.  In  2007,  both  the  SBIRS-High  and  AEHF  show  sub¬ 
stantial  cost  growth  (although  in  each  case,  the  programs  show  only  one  type  of  cost 
growth  and  so  appear  in  the  upper  left  and  lower  right  quadrants).  An  analyst  should 
be  concerned  about  both  programs  given  their  substantial  size.  Concern  about  the 
MUOS  program  might  also  be  warranted.  Finally  in  2012,  the  picture  seems  to  have 
changed  somewhat.  Now,  the  SBIRS-High  program  is  a  sizeable,  strong  performer 
with  negative  unit  cost  growth.  The  AEHF  program  also  shows  some  improvement, 
important  again  because  of  the  program’s  size.  The  GBS  program  is  another  strong 
performer  in  this  case  with  negative  percentage  change  in  APUC  and  PAUC  (although 
it  is  small).  However,  one  program  still  warrants  some  additional  concern,  namely,  the 
NAVSTAR  GPS  program. 

The  final  visualization  we  used  in  our  analysis  combines  six  different  metrics  into 
a  radar  plot,  also  described  in  Chapter  Eight.  Each  of  six  vertices  of  the  figure  repre¬ 
sents  a  different  metric,  and  the  shape  made  by  connecting  a  portfolio’s  status  on  each 
of  these  six  metrics  creates  a  shape  that  provides  an  aggregate  view  of  the  portfolio  in 
any  given  year.  The  six  vertices  represent  the  percentage  of  programs  with  a  5  percent 
or  more  unit  cost  growth  (PAUC  and  APUC),  the  percentage  of  programs  with  an 
APB  breach,  the  percentage  of  programs  with  a  Nunn-McCurdy  breach,  the  percent¬ 
age  of  programs  with  any  quantity  change  compared  to  the  prior  year,  and  the  percent¬ 
age  of  programs  pre-Milestone  C.  Because  one  of  the  vertices  compares  one  year  of  the 
portfolio  with  a  prior  year  (quantity),  we  can  present  only  10  charts  for  our  11  years  of 
data  (no  graph  is  shown  for  2002).  In  general,  a  smaller  shape  represents  better  perfor¬ 
mance  overall — lower  unit  cost  growth,  fewer  changes  in  quantity,  and  fewer  breaches. 
We  can  compare  the  portfolio’s  status  over  time  by  comparing  the  size  of  these  shapes 
in  each  year  of  assessment.  We  present  two  ways  of  doing  this  in  Figure  9.17,  which 
shows  the  radar  plots  for  the  helicopter  portfolio  in  each  of  the  ten  years  of  our  assess¬ 
ment.  Although  the  size  of  the  shape  in  the  radar  plot  increases  initially  though  about 
2009,  it  appears  to  shrink  gradually  after  2010.  Also  notable  from  this  figure  is  the 
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Figure  9.16 

Unit  Cost  Growth  and  Program  Size,  Satellite  Portfolio,  2002,  2007,  and  2012 
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Figure  9.17 

Aggregate  Portfolio  Performance,  Helicopter  Portfolio,  2002  Through  2012 
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fact  that  there  appears  to  be  an  increase  in  the  percentage  of  programs  experiencing 
quantity  changes  in  2011  and  2012.  However,  as  noted  elsewhere,  having  a  change  in 
quantity  procured  is  not  necessarily  a  sign  of  poor  program  or  portfolio  performance. 
Figure  9.18  shows  the  radar  plots  for  2003,  2007,  and  2012  on  a  single  chart  and  pres¬ 
ents  a  similar  view  of  the  evolution  of  portfolio  performance  over  time.  Although  the 
shape  for  2007  is  larger  than  that  for  2003,  the  shape  for  2012  is  considerably  smaller 
than  the  other  two.  This  would  be  consistent  with  the  story  told  by  many  of  the  indi¬ 
vidual  metrics:  There  is  an  apparent  improvement  in  the  portfolio’s  performance  after 
about  2008.  However,  our  analysis  thus  far  has  not  looked  into  why  we  see  this  change, 
what  factors  might  be  driving  it,  or  whether  it  represents  a  true  improvement  of  the 
entire  portfolio  or  simply  a  change  in  one  or  two  programs. 

Figures  9.19  and  9.20  show  the  radar  charts  for  the  satellite  portfolio,  first  for  all 
ten  years  and  then  for  2003,  2007,  and  2012  on  a  single  graph.  The  ten-year  compari¬ 
son  shows  a  shape  that  appears  to  change  and  increase  in  size  over  the  period  2003  to 
2010  before  shrinking  in  2011  and  2012.  It  is  interesting  that  the  changes  in  the  radar 
charts  for  the  satellite  portfolio  are  changes  not  only  in  size  but  also  in  shape.  This  sug¬ 
gests  that  the  status  of  the  portfolio  on  the  six  different  metrics  changed  in  different 
ways  in  the  ten  years  of  our  analysis.  This  is  consistent  with  our  general  observation 
that  the  composition  of  the  satellite  portfolio  appears  to  change  more  often  over  time. 
Future  assessments  can  consider  the  reasons  for  the  changing  shapes  of  these  radar 
plots  in  more  detail.  Finally,  Figure  9.20  compares  shapes  for  years  2003,  2007,  and 
2012.  Although  the  shape  for  2007  is  somewhat  larger  than  that  for  2003  and  2012, 
the  shapes  for  2003  and  2012  are  more  similar.  This  pattern  is  similar  to  that  observed 
for  other  metrics,  namely,  a  satellite  portfolio  that  had  several  years  of  declining  per- 


Figure  9.18 

Aggregate  Portfolio  Performance,  Helicopter 
Portfolio,  2003,  2007,  and  2012 
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Figure  9.19 

Aggregate  Portfolio  Performance,  Satellite  Portfolio,  2002  Through  2012 
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Figure  9.20 

Aggregate  Portfolio  Performance,  Satellite 
Portfolio,  2003,  2007,  and  2012 
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formance  between  2002  and  2012  but  that  improved  somewhat  in  recent  years.  A  fol- 
low-on  assessment  of  the  portfolio  could  explore  the  drivers  of  this  pattern,  including 
specific  programs  responsible  for  the  overall  trend  and  external  triggers  or  events  that 
might  explain  the  observed  changes  over  time. 


Chapter  Summary 

This  section  applied  our  methodology  and  chosen  visualizations  for  portfolio  analy¬ 
sis,  laid  out  in  detail  in  Chapter  Eight,  to  two  real  portfolios,  helicopter  and  satellite. 
This  demonstration  included  descriptors  such  as  percentage  of  programs  at  Milestones 
B  and  C,  number  and  type  of  breaches,  indicators  to  include  unit  cost,  percentage 
of  funds  remaining,  and  percentage  of  programs  with  any  quantity  change.  We  also 
looked  at  multimetric  charts  that  provide  a  more  aggregate  perspective  on  the  perfor¬ 
mance  of  the  two  portfolios.  The  most  significant  similarity  between  the  two  portfolios 
is  that  both  seem  be  improving  on  certain  metrics  and  at  the  aggregate  level  in  recent 
years.  The  two  portfolios  show  some  differences  as  well  as  some  similarities.  The  heli¬ 
copter  portfolio  appears  to  show  a  gradually  maturing  (and  possibly  aging)  portfolio 
that  experiences  some  improvement  in  unit  cost  growth  and  number  of  breaches  over 
the  assessment  period.  The  satellite  portfolio,  on  the  other  hand,  has  more  year-to-year 
changes  in  composition  and  somewhat  less  change  in  overall  portfolio  status  over  the 
assessment  window.  However,  the  satellite  portfolio  still  has  some  programs  that  expe¬ 
rience  extended  periods  of  poor  performance  especially  in  the  years  between  2007  and 
2010. 
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This  chapter  focused  primarily  on  describing  the  visualizations  used  for  an  analy¬ 
sis  of  either  portfolio  and  did  not  conduct  a  full  assessment  of  either.  One  important 
lesson  here  is  the  importance  of  viewing  multiple  metrics  from  different  perspectives  to 
better  understand  patterns  and  trends  of  portfolio  status  and  future  risk.  Single  metrics 
do  not  provide  a  complete  picture  of  portfolio  performance.  The  interpretation  of  per¬ 
formance  metrics  must  also  be  done  in  the  context  of  the  descriptive  metrics,  especially 
awareness  of  programs  moving  in  and  out  of  the  portfolio,  the  potential  for  one  or  two 
programs  to  dominate  the  results  of  portfolio  metrics,  and  the  size  of  the  program  and 
funds  remaining  as  a  measure  of  future  portfolio  risk.  In  addition,  the  kind  of  analyses 
demonstrated  in  this  chapter  can  identify  results  that  can  then  be  explored  in  more 
detail.  In  the  next  chapter,  we  go  one  step  further  in  providing  a  sample  expanded 
narrative  of  the  helicopter  portfolio  based  on  our  observations  from  the  metrics  and 
visualizations  shown  here. 


CHAPTER  TEN 


Expanded  Narrative  of  the  Helicopter  Portfolio 


The  previous  chapter  outlined  an  initial  portfolio  analysis  presentation,  which  visual¬ 
izes  the  “what”  of  a  portfolio’s  status  in  the  context  of  the  analysis’s  initial  objectives 
(in  our  case,  cost  and  schedule  risk  indicators).  The  analysis  and  visualizations  allow 
for  a  repeatable  and  auditable  process  to  track  portfolio  status  indicators  over  time 
or  across  various  portfolios.  Inevitably,  however,  a  policymaker  or  decisionmaker  will 
want  to  delve  into  “why”  portfolio  trends  or  patterns  have  emerged  to  adjust  his  or 
her  policies  or  priorities  accordingly,  and  to  understand  whether  near-term  action  can 
mitigate  future  portfolio  risk.  This  examination  can  take  many  forms  depending  on 
the  questions  asked.  If  certain  programs  appear  to  be  the  driving  force  behind  overall 
portfolio  trends,  a  program-based  root  cause  analysis  may  be  required.1  Other  ques¬ 
tions  may  require  revisiting  the  portfolio  analysis  objectives  to  broaden  the  overall  data 
and  metrics  collection  needs.  In  this  chapter,  we  provide  an  example  narrative  that 
arose  from  questions  inspired  from  observed  unusual  or  interesting  trends  within  the 
helicopter  portfolio  analysis  results.  Although  we  did  not  conduct  a  comprehensive 
root  cause  analysis  for  each  program  in  the  portfolio,  we  did  use  general  root  cause 
methods  to  pinpoint  cost  growth  events  for  the  portfolio  and  to  identify  both  the  indi¬ 
vidual  program  performance  issues  and  external  forces  or  trends  that  may  have  driven 
those  events.  The  major  categories  of  potential  policymakers’  questions  include  greater 
contextual  understanding,  further  examination  into  the  portfolio  analysis  cost  and 
schedule  risk  objectives,  and  further  potential  analysis  inspired  by  the  observed  trends 
within  the  portfolio  analysis. 


1  For  a  comprehensive  methodology  on  performing  root  cause  analyses  of  defense  acquisition  programs,  see  Irv 
Blickstein,  Jeffrey  A.  Drezner,  Brian  Mclnnis,  Megan  McKernan,  Charles  Nemfakos,  Jerry  M.  Sollinger,  and 
Carolyn  Wong,  Methodologies  in  Analyzing  the  Root  Causes  of  Nunn-McCurdy  Breaches,  Santa  Monica,  Calif.: 
RAND  Corporation,  TR-1248-OSD,  2012. 
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Providing  Further  Context 

Although  the  descriptive  metrics  provide  a  general  context  for  many  of  the  perfor¬ 
mance  metrics,  additional  details  may  be  required  to  provide  a  greater,  more  nuanced 
picture.  For  example,  what  constitutes  poor  performance  in  terms  of  cost  growth  for 
a  portfolio  with  many  pre-Milestone  C  developmental  programs  may  be  different  for 
a  portfolio  with  many  programs  that  have  already  entered  full-rate  production.  The 
helicopter  portfolio  included  13  programs  reporting  between  the  years  2002  and  2012. 
During  that  time,  some  programs  were  initiated,  others  cancelled,  and  others  finished 
reporting.  Only  five  of  the  programs  were  actively  reporting  for  the  entire  period. 
For  this  reason,  many  of  our  metrics  are  discussed  as  a  percentage  of  programs  in  the 
portfolio  that  carry  specific  attributes.  Key  attributes  of  this  portfolio  are  displayed  in 
Table  10.1  and  include  the  following: 

•  The  portfolio  contains  a  mix  of  force  application  (73  percent)  and  logistics 
(27  percent)  platforms. 

•  The  majority  of  active  programs  in  the  portfolio  were  mature  (past  Milestone  C) 
by  2012. 

•  Army-  and  Navy-managed  programs  only  are  included  (no  Air  Force-managed 
programs  reported). 

•  Three  programs  with  developmental  and  modified  COTS  acquisition  strategies 
were  cancelled  during  the  observed  time  period. 

•  All  actively  reporting  programs  are  modifications,  upgrades,  or  evolutionary 
increments  of  existing  military  airframes. 

Over  time,  the  helicopter  portfolio  composition  shifted  from  mostly  force  appli¬ 
cation  to  more  logistics  programs  (see  Table  9.1  for  a  view  of  the  changing  composition 
of  the  portfolio  over  time).  In  part,  this  shift  may  have  resulted  because  the  majority  of 
the  Army’s  rotary  aviation  portfolio  now  resides  in  the  utility  and  cargo  aviation  fleets, 
and  future  plans  for  the  portfolio  are  concentrated  on  sustainment,  recapitalization, 
remanufacturing,  and  incremental  modernization  of  existing  fleets.  Overall,  DoD  has 
only  three  new  aircraft  slated  for  production  in  the  foreseeable  future:  the  MH-XX 
replacement  for  MH-60R  and  S  variants,  the  VXX  presidential  helicopter,  and  the 
Armed  Aerial  Surveillance  (AAS),  a  replacement  for  the  OH-58D  Kiowa  Warrior  that 
is  currently  a  pre-MDAP.2 


2  For  additional  details,  see  “Annual  Aviation  Inventory  and  Funding  Plan,  Fiscal  Years  (FY)  2014-2043,”  May 
2013.  Notably,  the  AAS  will  be  part  of  the  Army’s  Future  Vertical  Lift  program.  The  program  will  develop  four 
different  size  aircraft  that  will  share  common  sensors,  hardware,  and  other  technology.  The  new  aircraft  will 
replace  the  Army’s  UH-60  Black  Hawk,  AH-64  Apache,  CH-47  Chinook,  and  OH-58  Kiowa  helicopters.  See 
“U.S.  Army  Future  Vertical  Lift  Helicopter  Reinvention  Prototypes  Will  Be  Flying  in  2017,”  Next  Big  Future, 
January  31,  2015. 


Table  10.1 

Helicopter  Portfolio  Overview 


Program  Name 

Service 

Program  Type 

Helicopter  Type 

Status  as  of  2012 

Comanche 

Army 

Developmental 

Force  application 

Cancelled  (2003)  because  of  Army  decision  that  the  aircraft  no  longer 
met  force  requirements;  costs  were  also  an  issue 

VH-71 

Navy 

Modified  COTS  (AgustaWestland  AW101) 

Force  application 

Cancelled  (2009  because  of  poor  cost  and  schedule  performance 

ARH 

Army 

Modified  COTS  (Bell  407)  with  nondevelop- 
mental  mission  equipment  packages 

Force  application 

Cancelled  (2008)  because  of  poor  cost  performance  and  test  failures 

Longbow  Apache 

Army 

Modification/upgrade  of  existing  AH-64 

Force  application 

90  percent  complete,  stopped  reporting  in  2011 

LUH 

Army 

COTS  (non-developmental) 

Logistics 

90  percent  complete,  stopped  reporting  in  2012 

UH-60M  Upgrades 

Army 

Modification/upgrade  of  UH-60 

Logistics 

Milestone  C  (2001) 

MH-60S 

Navy 

Modification/upgrade  of  SH-60B 

Force  application 

Milestone  C  (2002) 

MH-60R 

Navy 

Modification/upgrade  of  SH-60B 

Force  application 

Milestone  C  (2006) 

CH-47F 

Army 

Modification/upgrade  of  CH-47D 

Logistics 

Milestone  C  (2004) 

H-1  Upgrades 

Navy 

Modification/upgrade  of  UH-1Y  and  AH-1Z 

Force  application 

Milestone  C  (2008) 

CH-53K 

Navy 

Evolutionary  increment  of  CH-53E 

Logistics 

Milestone  B  (2005) 

AH-64E  Re¬ 
manufacture 

Army 

Modification/upgrade  of  AH-64 

Force  application 

Milestone  C  (2010) 

AH-64E  New  Build 

Army 

Modification/upgrade  of  AH-64 

Force  application 

Milestone  C  (2010) 

VXX/VH-92 

Navy 

Variant  of  H-92  Superhawk  military  transport 

Force  application 

Future  program;  contract  award  for  development  in  2014 

AAS 

Army 

Replacement  or  upgrade  for  OH-58D  Kiowa 
Warrior 

Force  application 

Pre-MDAP 

MH-XX 

Navy 

Replacement  for  the  MH-60R  and  S 

Force  application 

Future  program;  capabilities-based  assessment  held  2013 

NOTES:  The  gray-shaded  programs  in  the  top  five  rows  are  those  that  are  either  greater  than  90  percent  complete  (and  so  no  longer  reporting)  or 
were  cancelled.  The  white  programs  are  current,  active  programs.  The  three  orange  rows  at  the  bottom  of  the  table  are  developmental  and  pre-MDAP 
programs. 
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Overall,  judging  by  these  attributes,  we  can  characterize  the  current  active  heli¬ 
copter  portfolio  as  a  mature  portfolio  with  a  reasonably  well-diversified  contracting 
base  (having  more  than  a  single  contractor  involved  in  the  procurement  of  different 
aircraft  in  the  portfolio  can  lower  procurement  risk  compared  to  dependence  on  single 
source)  and  fairly  low  technical  and  contract  risk.3  The  increasing  maturity  of  the  over¬ 
all  portfolio  is  due  to  the  maturing  of  the  programs  in  existence,  no  new  entrants,  and 
termination  of  several  of  the  newer  entrants  (ARH  and  VH-71).  The  only  Milestone  B 
program  is  CH-53K.  Although  this  indicates  more  stability  for  the  portfolio,  portfolio 
maturity  may  also  be  problematic  because  it  suggests  a  lack  of  new  development,  which 
could  be  a  source  of  risk  in  the  future.  It  is  worth  noting  that  the  CH-53K,  although 
technically  an  evolutionary  program,  is  significantly  different  from  its  predecessor, 
adding  some  new  risk  but  also  new  development. 


Spending  Trends 

One  metric  that  can  provide  context  into  the  evolution  of  a  portfolio  over  time  is  spend¬ 
ing  trends.  Spending  rates  may  be  relevant  to  a  portfolio  analysis  for  a  few  reasons.  If 
spending  rates  are  steep  but  there  is  no  evidence  of  any  other  technical  problem,  then 
this  may  not  be  a  signal  for  concern.  However,  in  some  cases,  steep  spending  rates  may 
be  a  sign  of  underlying  technical  problems  that  must  be  addressed.  In  other  cases,  rapid 
spending  may  be  a  warning  sign  that  a  program  is  likely  to  run  out  of  funding  before 
completion.  Because  it  may  be  a  signal  of  current  or  future  problems,  programs  with 
high  rates  of  spending  may  warrant  additional  attention  or  investigation.  Figures  10.1 
and  10.2  show  the  percentage  remaining  for  both  RDT&E  and  procurement  funding, 
which  reveals  not  only  how  much  is  left  for  individual  programs  in  2012  but  also  the 
rate  at  which  they  are  spending.  These  figures  can  be  a  first  step  in  helping  answer  the 
following  potential  policymaker  questions: 


Percentage  of  RDT&E  Programs  seem  to  have  spent  most/all  of  their  RDT&E  funds.  Which  programs 
funds  remaining  still  have  some  funds  remaining? 

(Figure  10.1)  Do  we  |<now  anything  about  how  those  funds  will  be  used? 

Does  the  amount  of  remaining  RDT&E  funds  appear  adequate  given  the 
acquisition  phase  of  the  program? 

Which  programs  are  spending  at  a  faster  rate  and  why?  Does  this  indicate 
technical  problems? 


3  These  assumptions  are  based  on  evidence  in  the  literature  suggesting  that  evolutionary  acquisition  strategies 
(incremental  modifications  of  existing  platforms)  typically  have  lower  technical  risks  and  that  a  firm  fixed-price 
contract  places  more  of  the  cost  performance  risk  on  the  contractor. 
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Percent  procurement  Percentage  of  procurement  funds  remaining  appear  to  be  declining.  Which 
funds  remaining  programs  have  procurement  funds  remaining? 

(Figure  10.2)  Which  programs  seem  to  be  spending  much  faster/slower  than  average?  Do 

these  programs  share  any  other  characteristics? 

Are  the  programs  spending  in  a  manner  commensurate  with  quantity  of 
aircraft  (or  vehicles,  weapons,  etc.)  left  to  procure? 

Which  programs  have  spent  all  nearly  all  of  their  procurement  dollars,  and 
what  is  likely  to  happen  to  them  once  they  finish  spending  their  money? 


As  shown  in  Figure  10.1,  AFI-64E  Remanufacture,  CFI-53K,  H-l  Upgrades, 
MF4-60S,  and  MFI-60R  still  have  RDT&E  funds.  As  CEI-53K  is  still  in  EMD,  it  is 
not  surprising  that  it  has  almost  50  percent  of  its  RDT&E  funding  still  available.  The 
A1T-64E  Remanufacture  also  has  a  significant  portion  of  its  RDT&E  funding  left. 
Specifically,  the  program  received  additional  RDT&E  funds  in  2010  to  support  addi¬ 
tional  labor  costs  to  improve  the  AH-64E’s  combat  capability.4  The  other  three  pro¬ 
grams  have  far  less  RDT&E  funding  remaining  but  still  have  several  RDT&E  lines  of 
effort  as  of  2012.  A  few  other  programs  appear  to  have  received  new  RDT&E  funds 
midway  through  their  program  life  cycle.  For  example,  the  UFI-60M  and  MFI-60S 
received  additional  RDT&E  funds  in  2004.  For  the  MF4-60S,  these  additional 
funds  went  to  additional  testing  and  to  incorporate  some  additional  design  improve¬ 
ments.5  For  the  UH-60M,  the  additional  funds  supported  several  improvements  to  the 

Figure  10.1 

Percentage  of  RDT&E  Funding  Remaining,  by  Program,  2002  Through  2012 


2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012 


H-1  Upgrades 

Longbow 

Apache 

New  Build 

Remanufacture 

4  Department  of  Defense,  AH-64E  Remanufacture,  Selected  Acquisition  Report,  December  2010c. 

5  Department  of  Defense,  MH-60S,  Selected  Acquisition  Report,  2004a. 
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UH-60M,  including  Common  Avionics  Architecture  System  (CAAS),  Fly-By-Wire 
and  Full  Authority  Digital  Engine  Control.6  In  2008,  the  FI-1  Upgrades  also  received 
additional  RDT&E  funds,  which  went  toward  a  redesign  as  well  as  the  development 
and  testing  of  new  software.7  Finally,  the  MH-60R  experienced  a  smaller  increase  in 
RDT&E  funding  in  2006,  which  also  went  toward  design  improvements.8  Another 
potentially  important  insight  from  Figure  10.1  is  the  flatness  of  the  FI-1  Upgrades 
and  MH-60R  RDT&E  spending  curves.  These  could  be  considered  worrisome  trends, 
as  the  flat  curve  could  indicate  that  the  programs  are  not  conducting  the  necessary 
RDT&E  activities  for  which  the  money  was  allocated — a  signal  of  potential  program 
problems  or  schedule  delays.  An  analyst  noting  these  flat  curves  might  choose  to  inves¬ 
tigate  the  RDT&E  progress  of  these  two  programs  in  more  detail. 

Some  additional  consideration  of  ongoing  R&D  within  the  helicopter  portfolio 
is  useful  because  it  provides  insight  into  the  future  of  the  portfolio,  including  poten¬ 
tial  sources  of  risk  and  stability  going  forward.  For  example,  the  FI-1  Upgrades  will 
include  the  development  of  a  new  cabin.  However,  the  first  AH-1Z  New  Build  cabin 
is  reported  to  be  one  year  late  and  experiencing  cost  overruns,  which  may  signal  risk 
that  the  program  will  require  additional  fund  increases  from  its  uptick  in  2008.  The 
MH-60R  is  using  its  RDT&E  funds  to  increase  the  reliability  of  its  Airborne  Low 
Frequency  Sonar  and  continue  its  Automatic  Radar  Periscope  Detection  and  Discrimi¬ 
nation  SDD  program.  Finally,  the  MH-60S  is  developing  the  LAU-61G/A  Digital 
Rocket  Launcher  with  Advanced  Precision  Kill  Weapons  System  and  is  also  conduct¬ 
ing  tests  and  operational  assessments  of  the  AN/AES-1  Airborne  Laser  Mine  Detec¬ 
tion  System,  AN/AQS-235  Airborne  Mine  Neutralizer  System  and  Organic  Airborne 
Surface  Influence  Sweep  (OASIS).  In  March  2012,  the  CNO  ended  efforts  to  integrate 
the  OASIS  system  and  the  Sonar  Mine  Detection  Set  on  the  platform,  which  may  have 
freed  up  additional  funds  for  the  program. 

Figure  10.2  shows  the  rates  of  procurement  spending  for  all  programs.  As  of  2012, 
the  only  program  that  is  not  spending  procurement  dollars  (besides  the  programs  that 
have  been  completed  or  cancelled)  is  the  CH-53K  program,  which  is  in  development. 
As  shown  by  the  graph,  the  UH-60M,  H-l  Upgrades,  MH-60R,  Longbow,  AH-64E 
Remanufacture,  and  MH-60S  appear  to  be  spending  their  procurement  dollars  at  sim¬ 
ilar  rates,  even  though  they  differ  on  the  percentage  left  as  of  2012.  The  Lakota  light 
utility  helicopter  (LUH),  however,  is  spending  its  procurement  funds  at  a  faster  rate, 
but  is  nearing  program  completion.  An  analyst  might  be  concerned  about  this  faster 
rate  of  spending,  but  the  fact  that  the  program  is  close  to  completion  may  mitigate  this 
concern.  The  AH-64E  New  Build  also  appears  to  be  spending  its  procurement  funds 
somewhat  faster  than  other  programs,  but  only  really  in  the  past  year  or  so.  Since  the 


6  Department  of  Defense,  UH-60M,  Selected  Acquisition  Report,  2004b. 

7  Department  of  Defense,  H-l  Upgrades,  Selected  Acquisition  Report,  2008a. 

8  Department  of  Defense,  MH-60R ,  Selected  Acquisition  Report,  2006. 
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Figure  10.2 

Percentage  of  Procurement  Funding  Available,  by  Program,  2002  Through  2012 
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program  only  recently  passed  Milestone  C,  it  is  really  too  soon  to  tell  if  this  represents 
a  long-term  trend.  However,  if  this  faster  rate  of  spending  were  to  continue,  this  might 
be  a  signal  of  possible  problems,  since  it  could  compromise  the  ability  of  the  program 
to  meet  procurement  objectives.  To  fully  understand  the  implications  of  these  observa¬ 
tions,  an  analyst  would  need  to  conduct  a  more  complete  investigation  of  each  indi¬ 
vidual  program  than  we  do  in  this  report.  This  additional  analysis  would  still  be  part 
of  the  portfolio  analysis  process,  as  the  process  is  one  that  has  many  layers  and  that  can 
be  taken  as  far  as  analyst  needs  or  desires. 

Figure  10.3  shows  the  percentage  of  remaining  procurement  funding  versus 
remaining  program  time  according  to  the  anticipated  time  line  for  the  program.  This 
metric  is  primarily  an  indicator  of  the  need  for  more  in-depth  assessment.  Possible 
follow-on  questions  could  be: 

•  Is  the  remaining  funding  adequate  for  the  remaining  quantities? 

•  Is  there  sufficient  time  to  procure  the  remaining  quantities? 

In  addition  to  the  above  questions,  Figure  10.3  allows  us  to  make  several  obser¬ 
vations  about  the  rate  of  spending  of  various  programs  within  the  portfolio  that  were 
active  as  of  2012.  Only  two  programs,  the  MH-60S  and  AH-64E  New  Build,  have 
a  smaller  percentage  of  procurement  funds  remaining  than  they  do  time  remaining. 
This  may  be  concerning,  since  it  suggests  that  these  programs  may  run  out  of  funding 
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Figure  10.3 

Percentage  of  Procurement  Funding  Remaining  Versus  percentage  of  Time  to  Program 
Completion 
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before  procurement  is  complete.9  For  the  AH-64E  New  Build,  the  large  percentage  of 
both  funding  and  time  remaining  suggests  that  it  is  too  soon  to  be  overly  concerned 
about  the  status  of  this  program.  For  the  MH-60S,  the  difference  between  percent¬ 
age  of  time  and  percentage  of  procurement  funds  remaining  is  small  and  the  program 
is  nearing  completion,  again  tempering  concern.  For  other  programs  in  the  portfo¬ 
lio,  the  percentage  of  funds  remaining  greatly  exceeds  the  percentage  of  time  remain¬ 
ing.  For  example,  the  CH-53K  has  100  percent  of  its  procurement  funds  remaining 
but  only  60  percent  of  its  time  remaining.  This  difference  may  raise  the  question  of 
whether  the  program  will  be  able  to  spend  all  its  funds  and  procure  needed  quantities 
before  its  planned  end  date. 

Deeper  Examination  into  Cost  and  Schedule  Risk:  Unit  Cost  and  Breaches 

Over  the  11 -year  period  for  which  we  gathered  data,  the  overall  status  of  the  port¬ 
folio  seems  to  improve  along  the  dimensions  considered  by  our  portfolio  metrics. 


9  Of  course,  there  is  less  concern  that  the  project  will  end  with  funds  remaining,  since  if  funding  remains  when 
a  program  is  near  completion,  it  can  more  easily  be  reallocated.  That  said,  a  rate  of  spending  that  is  too  slow  may 
also  be  worrisome,  if  it  suggests  that  a  program  is  not  going  to  meet  schedule  deadlines  or  that  it  will  leave  unused 
money  on  the  table. 
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Specifically,  we  observe  some  improvements  in  our  portfolio  metrics  between  2008  and 
2009  and  2012.  In  particular,  average  portfolio  cost  growth  has  decreased  significantly. 
However,  as  we  will  discuss  more  in  our  analysis  of  the  portfolio,  although  the  helicop¬ 
ter  portfolio  does  show  some  improvement  in  terms  of  unit  cost  growth  and  schedule, 
a  complete  portfolio  assessment  considers  the  portfolio’s  performance  on  many  dimen¬ 
sions,  not  just  one  or  two.  Through  our  measurement  and  visualizations  of  portfolio 
indicators,  we  identified  the  following  areas  and  questions  for  further  analysis: 


Unit  cost  growth  What  programs  are  driving  high  unit  cost  growth  between  2007  and  2009? 

Why  does  unit  cost  growth  for  the  portfolio  appear  to  decline  over  time  and 
what  is  driving  that  decline? 

What  factors  at  the  program-level  appear  to  be  driving  program  outcomes  and 
how  does  this  affect  our  assessment  at  the  portfolio  level? 

What  is  the  effect  of  rebaselining  on  this  metric? 

What  is  the  relationship  between  unit  cost  growth  and  other  indicators? 

Breaches  Which  programs  have  the  most  breaches  and  what  types?  Are  there  any  patterns 

in  breach  types  (e.g.,  breach  types  that  seem  to  affect  all  programs  or  no 
programs?) 

Are  there  any  patterns  in  root  causes  or  drivers  of  observed  breaches? 

Multimetrics  What  drove  the  large  shape  of  the  radar  chart  in  2004?  How  was  this  corrected  in 

2005? 

Why  does  the  performance/shape  size  appear  to  grow  in  2006,  2007,  and  2008? 

What  program  or  programs  helped  turn  this  around  starting  in  2010?  How  were 
these  changes  accomplished?  Was  it  because  programs  ended?  Policy  changes? 
Rebaselines? 

Comparing  2003,  2007,  and  2012:  What  explains  the  very  different  shape  in  2003? 
How  can  we  compare  this  to  2007  and  2012 


Unit  Cost  Growth  and  the  Effects  of  Rebaselining 

In  some  years  over  the  time  period  observed,  the  portfolio  appeared  to  have  higher  than 
average  unit  cost  growth;  for  example,  changes  in  APUC  and  PAUC  between  2007 
and  2012  were  statistically  significant,  and  percentage  unit  cost  growth  in  2008  and 
2009  was  statistically  different  and  higher  than  cost  growth  in  other  years.  To  under¬ 
stand  the  causes  of  this  cost  growth,  we  investigated  the  performance  of  individual 
programs.  The  difference  between  2008-2009  and  2012  may  even  be  underestimated 
in  our  data  because  the  VH-71  program  was  reporting  low  estimated  cost  growth  in 
the  SARs  and  DAES;  however,  government  changes  to  capability  requirements  led  to 
a  restructured  total  program  estimate  that  was  nearly  twice  the  baseline  estimate.  The 
VH-71  negative  cost  growth  reported  in  2008,  which  is  interpreted  as  strong  cost  per¬ 
formance  if  only  viewed  from  the  perspective  of  a  single  metric,  results  only  because 
of  an  increase  in  quantity  without  a  parallel  increase  in  funding.  The  VH-71  program 
was  subsequently  cancelled  in  2009  under  heavy  political  pressure.10 


10  Senator  John  McCain,  for  example,  was  vocal  in  his  frustration  at  the  program’s  rising  costs.  “We  can’t  afford,” 
he  asserted  on  February  24,  “to  have  a  helicopter  built  for  the  president  of  the  United  States  that  costs  more  than 
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The  ARH  program,  which  suffered  from  contractor  management  issues,  test  fail¬ 
ures,  delays,  and  cost  growth  in  excess  of  40  percent,  was  also  driving  some  of  the  high 
portfolio  unit  cost  growth  in  the  2008-2009  time  period.  Its  cancellation  in  2009 
helped  to  improve  the  overall  portfolio  status  with  regards  to  the  cost  metrics. 

The  IT-l  Upgrades  program  was  also  performing  poorly  in  2008  and  suffered  an 
APUC  breach  above  15  percent;  however,  this  program  was  not  subsequently  cancelled 
but  instead  was  rebaselined.  Unlike  the  ARH  and  VH-71  programs,  internal  perfor¬ 
mance  issues  were  not  the  main  reasons  for  high  cost  growth  for  the  H-l  Upgrades. 
Rather  than  significant  program  performance  issues,  there  were  external  demand 
forces  driving  the  changes  to  this  program  and  necessitating  a  new  baseline  cost  esti¬ 
mate.  The  main  driving  force  was  changes  to  the  force  structure  needs  of  the  Marine 
Corps,  requiring  additional  aircraft  quantities.  The  existing  inventory  of  AH-1Z  avail¬ 
able  for  remanufacture  was  less  than  needed  for  conversion,  and  thus  the  program  was 
restructured  to  build  entirely  new  platforms  to  meet  the  inventory  shortfall,  according 
to  its  SARs.  This  example  highlights  the  caution  that  must  be  taken  in  interpreting 
some  of  the  cost  growth  metrics. 

Finally,  the  AH-64E  Remanufacture  suffered  a  high  unit  cost  growth  and  an 
APB  breach  for  both  APUC  and  PAUC  in  2009.  In  this  case,  however,  the  breach 
was  less  a  performance-driven  issue  than  a  change  in  how  costs  for  the  program  were 
reported.  This  change  resulted  in  a  high  unit  cost  growth  in  that  year  but  not  a  sys¬ 
temic  or  lasting  problem.  Subsequent  improvement  in  the  program  contributed  to  the 
overall  improved  cost-growth  performance  of  the  helicopter  portfolio  in  later  years. 

To  reiterate  a  point  made  above,  however,  one  must  not  focus  on  a  single  metric 
without  understanding  its  context  and  how  the  particular  metric  is  calculated.  At  the 
portfolio  level  over  the  entire  time  period,  unit  cost  growth  appears  to  be  lower  in  2012 
than  in  2002  with  more  significant  changes  when  we  compare  2008  and  2009  to  2012. 
However,  although  portfolio  unit  cost  growth,  when  compared  to  the  original  baseline, 
appears  to  decline  between  2002  and  2012,  this  may  not  reflect  an  actual  improvement 
in  unit  cost  growth  performance  because  of  rebaselines  that  also  occur  over  this  same 
period.  Table  10.2  shows  the  rebaselines  that  occurred  from  2002  to  2012.  In  fact, 
nearly  all  of  the  programs  had  at  least  one  change  to  their  current  baseline  during  the 
time  period  observed,  and  about  half  of  the  programs  also  had  a  change  to  their  origi¬ 
nal  baseline,  which  can  further  affect  apparent  changes  in  unit  cost  growth.  In  fact, 
the  MH-60R,  UH-60M,  and  the  H-l  Upgrades  programs  have  all  had  four  or  more 
baseline  changes  over  the  10-year  period. 

The  decision  to  establish  a  new  baseline  may  follow  both  positive  and  negative 
program  events  and  changes  including  APB  breaches,  revised  cost  estimates  at  mile- 


Air  Force  One.”  Christopher  J.  Castelli,  “Navy,  Lockheed  Mum  Amid  Revelations  of  Soaring  VH-71  Costs,” 
Inside  the  Navy,  March  9,  2009.  The  program  was  cancelled  after  a  comprehensive  program  review  in  FY 
2010.  For  further  information,  see  Dan  Taylor,  “Navy  Issues  Stop-Work  Order  on  VH-71  Presidential  Helo,” 
DefenseAlert ,  May  15,  2009. 


Table  10.2 

Program  Current  Baseline  Changes,  2003  Through  2012 


2003 

2004 

2005  2006 

2007 

2008  2009  2010 

2011 

2012 

Total  Re¬ 
baselines 
(Current) 

H-1  Upgrades 

APB  breachers 

APB  breach/ 
restructure 

Nunn-McCurdy  NewSCP 

breach  cleared/ 
quantity  change 

4 

MH-60R 

New  cost 
estimate 

Changed 

procurement 

strategy 

APB  breach/ 
requirements 
change 

APB  breaches/ 
quantity  change 

5 

CH-47F 

Breach 

1 

MH-605 

APB  breaches 

Nunn-McCurdy 
breach  cleared/new 
cost  estimate 

2 

CH-53K 

Pre- 

Milestone  B 

New  cost 
estimates 

1 

LUH 

Pre-Milestone  B 

Contract 

change 

1 

UH-60M 

Changed 

procurement 

strategy 

APB  breaches 

Added  upgrade 

Increased 

quantity 

4 

Comanche 

New  ICE 

Cancelled 

1 

Longbow 

Requirements 

changes 

Stopped  reporting 

1 

ARH 

Pre- 

Milestone  B 

Cancelled 

0 

VH-71 

Pre- 

Milestone  B 

Cancelled 

0 

AH-64E 

remanufacture 

Pre-Milestone  B 

Program  splits  into 
remanufacture  and 
new  build 

1 

AH-64E  new  build 

Pre-Milestone  B 

0 

No.  f  programs 
rebaselined 

3 

2 

3  2 

2 

1  1  5 

1 

1 

21 

Expanded  Narrative  of  the  Helicopter  Portfolio  119 


120  Expanded  Narrative  of  the  Helicopter  Portfolio 


stone  approvals,  changes  to  the  contracting  or  procurement  strategy,  or  changes  to  the 
estimated  quantities.11  In  the  helicopter  portfolio,  half  of  the  programs  that  issued  a 
new  baseline  during  the  time  period  observed  did  so  (as  required)  in  response  to  either 
a  Nunn-McCurdy  or  an  APB  breach.  However,  the  other  half  of  observed  rebaselines 
occurred  without  any  precipitating  breach.12 

Rebaselining  has  the  effect  of  making  unit  cost  growth  look  lower  because  it 
raises  the  baseline  estimate.  When  looking  at  the  portfolio  as  a  whole  using  the  cur¬ 
rent  baseline  as  the  evaluative  metric,  unit  cost  performance  is  likely  to  show  signs  of 
improved  performance  in  years  following  a  large  percentage  of  program  rebaselines. 
For  example,  between  2010  and  2012,  seven  of  the  active  programs  in  the  portfolio  (10 
in  2010  and  nine  in  2011  and  2012)  issued  a  new  current  baseline.  It  is  important  to 
note  that  these  rebaselines  occurred  over  the  same  period  in  which  we  observed  declin¬ 
ing  percentage  unit  cost  growth  in  both  the  APUC  and  PAUC.  The  rebaselines  in  this 
period  may  contribute  to  a  misleading  view  of  unit  cost  growth.  Using  the  original 
baseline  to  calculate  unit  cost  growth  in  cases  where  rebaselines  have  occurred  provides 
a  more  accurate  picture  of  unit  cost  growth.  Looking  at  the  same  2010  to  2012  period 
using  the  original  baseline,  for  example,  will  tell  a  somewhat  different  story  about  unit 
cost  growth  than  the  current  baseline  graphs  shown  previously.  Figure  10.4  shows  the 
trend  in  unit  cost  growth  using  the  original  baseline.  Although  the  current  baseline 
chart  shown  in  Chapter  Nine  shows  a  decline  in  unit  cost  growth  in  recent  years,  the 
original  baseline  graph  does  not  show  the  same  trend.  Instead  the  chart  suggests  that 
the  PAUC  for  the  portfolio  as  a  whole  has  remained  largely  the  same.  The  median 
does  not  appear  to  change  much,  although  the  range  does  vary  from  year  to  year.  In 
this  case,  the  best  cost-growth  performance  appears  to  be  in  2005  and  2006,  with 
2007  through  2011  slightly  higher  (and  these  differences  are  statistically  significant  at 
p  <  0.01).  However,  the  original  baseline  chart  still  tells  a  “good  news”  story  about  the 
performance  of  the  portfolio.  Clearly,  the  percentage  unit  cost  growth  of  the  portfolio 
has  improved  since  the  period  from  2002  to  2004.  It  has  also  been  remarkably  steady 
in  recent  years,  even  when  the  beneficial  effects  of  rebaselining  are  factored  in.  This 
graph  could  also  be  used  to  allay  the  policymaker  fears  about  runaway  cost  growth,  at 
least  in  this  portfolio  program. 

The  discussion  of  the  percentage  unit  cost  growth  performance  of  the  portfolio 
measured  from  the  original  baseline  suggests  that  it  is  important  to  consider  both  indi¬ 
cators  when  evaluating  portfolio  risk.  An  analyst  should  expect  to  see  low  portfolio 


11  A  program  may  have  a  breach  or  quantity  changes  from  the  current  baseline  for  more  than  one  year  before 
restructuring  the  program  and  issuing  a  new  baseline. 

12  According  to  DoD  Instruction  5000.02  “Operation  of  the  Defense  Acquisition  System,”  January  7,  2015,  Pro¬ 
grams  may  rebaseline  under  the  following  circumstances:  At  milestone,  full-rate  production,  and  full  deployment 
decisions;  a  major  program  structure  approved  and  fully  funded  by  the  Milestone  Decision  Authority  (MDA);  or 
if  the  MDA  concludes  that  “fact-of-life  program  changes”  deem  management  to  the  existing  baseline  impractical. 
Programs  are  explicitly  prohibited  from  rebaselining  simply  in  order  to  avoid  a  breach. 
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Figure  10.4 

Percentage  Unit  Cost  Growth,  PAUC  Original  Baseline,  Helicopter  Portfolio,  2002 
Through  2012 


2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012 

Number  of 

programs  7  7  6  911  11  11  9  10  9  9 


NOTES:  Changes  between  2004  and  2005-2010  along  with  changes  from  2005  to 
2009-2011  are  statistically  significant  at  p  <  0.05.  MH-60S  omitted  2005-  2007,  2012;  ARH 
omitted  2008;  VH-71  omitted  2006-2007;  AH-64E  Remanufacture  omitted  2010  and 
2012. 
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unit  cost  growth  using  the  current  baseline  following  a  period  where  many  programs 
have  restructured  to  new  current  baselines.  The  analyst  will  not  see  this  same  improve¬ 
ment  and  will  get  a  more  accurate  picture  of  performance  on  unit  cost  growth  when 
the  original  baseline  is  used  to  calculate  unit  cost.  Although  it  may  be  misleading, 
looking  at  unit  cost  growth  using  current  baselines  can  still  be  valuable  for  an  analyst. 
For  example,  the  fact  that  restructuring  and  rebaselining  has  occurred  may  remove 
some  program  risk  if  funding  is  better  aligned  with  quantity  changes  or  new  cost  and 
schedule  estimates.  On  the  other  hand,  high  portfolio  unit  cost  following  a  period 
where  many  programs  have  restructured  to  a  new  baseline  may  be  a  warning  sign  that 
portfolio  performance  is  decreasing. 

It  is  difficult  to  identify  root  causes  for  unit  cost  growth  across  the  portfolio, 
especially  as  related  to  program  management.  There  may  potentially  be  a  relationship 
between  lower  cost  growth  and  maturation  of  programs  in  the  portfolio  with  many  in 
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production  and  few  in  development  by  2012.  However,  we  did  not  observe  a  relation¬ 
ship  within  this  set  of  data. 

We  would  assume  that,  in  general,  modifications  or  increments  of  existing 
platforms  should  have  lower  risk  than  revolutionary  development  programs  across  a 
number  of  technical  and  management  dimensions.  Only  one  program  in  the  helicop¬ 
ter  portfolio  (Comanche)  could  be  a  considered  a  new  development  program.  All  the 
other  programs  in  the  portfolio  are  modifications  of  existing  military  or  commercial 
platforms,  or  increments/evolutionary  developments  of  existing  helicopter  programs. 
In  fact,  platforms  in  the  CH-53  and  CH-47  family  have  been  in  production  since 
the  1960s.  However,  two  of  the  programs  that  were  cancelled  for  poor  cost  perfor¬ 
mance  (VH-71  and  ARH)  were  modifications  of  COTS  platforms.  These  programs 
may  have  still  been  “developmental”  in  terms  of  the  unproven  integration  of  different 
mission  packages/systems  into  a  commercial  platform.  Although  other  programs  are 
increments,  they  still  incorporate  new  developmental  technologies.  In  the  case  of  the 
CH-53K,  the  changes  planned  compared  to  its  predecessor  are  significant.  The  new 
helicopter  is  planned  to  have  twice  the  lift  capacity  of  the  previous  variant  and  makes 
other  advances  as  well.  Thus,  although  it  is  an  evolutionary  program,  there  is  still  much 
new  development  (and  therefore  time  and  risk)  involved. 


APB  and  Nunn-McCurdy  Breach  Trends 

The  collection  of  APB  and  Nunn-McCurdy  breach  data  provides  a  key  example  into 
why  the  context  of  current  policy  can  affect  a  metric’s  outcomes.  During  our  portfo¬ 
lio’s  time  frame,  the  policy  on  how  Nunn-McCurdy  breaches  were  defined  and  thus 
reported  changed.  In  the  FY  2006  NDAA  (P.L.  109-163),  DoD  was  directed  to  mea¬ 
sure  cost  growth  against  the  original  baseline  in  addition  to  the  current  baseline,  which 
it  has  traditionally  been  measured  against.  Because  of  this  reporting  change,  more 
programs  generally  incurred  more  Nunn-McCurdy  breaches  post-2006.  The  rules  one 
uses  for  counting  these  breaches  pre-  and  post-2006  greatly  influence  the  results.13  For 
our  analysis,  we  simply  recorded  the  Nunn-McCurdy  breaches  as  they  were  reported 
within  the  SARs  to  reflect  policy  priorities  at  the  time.  Other  analysts  may  wish  to  nor¬ 
malize  the  definition  of  Nunn-McCurdy  breach  and  perform  additional  calculations. 
In  addition,  another  key  issue  when  recording  Nunn-McCurdy  breaches  over  time  is 
that  programs  may  report  the  same  breach  for  multiple  years  until  it  has  been  restruc¬ 
tured.  We  took  this  into  account  in  our  analysis.  For  APB  breaches,  we  similarly  record 
them  as  they  were  reported  in  the  SARs  (e.g.,  if  the  SARs  reported  an  APB  breach,  we 


13  For  additional  discussion  on  this  issue  and  a  proposed  methodology  for  normalization  of  Nunn-McCurdy 
breach  data  collection,  see  Arena  et  ah,  2014a,  pp.  59-105. 


Expanded  Narrative  of  the  Helicopter  Portfolio  123 


do  likewise  here).14  For  the  helicopter  portfolio,  the  reduction  in  the  number  of  both 
APB  and  Nunn-McCurdy  breaches  is  statistically  significant  when  comparing  2012 
with  2007  and  2008  but  not  when  comparing  2002  and  2012.  As  discussed  in  the  sec¬ 
tion  above  on  unit  cost,  program  breaches  may  drive  a  restructuring,  rebaselining,  or 
cancellation  of  the  program  (more  likely  in  the  case  of  a  Nunn-McCurdy  breach).  The 
cancellation  of  the  ARH  and  VH-71  programs  by  2009  and  the  restructuring  of  four 
other  programs  in  2010  likely  contributed  to  a  reduction  in  breaches. 

When  we  looked  for  patterns  in  the  types  of  breaches  within  the  portfolio  over 
this  time  period,  we  found  that  the  most  common  type  of  breaches  were  O&S  or 
schedule  breaches,  accounting  for  half  of  the  total  breaches  as  Figure  10.5  shows.  Cost 
breaches  (PAUC  and  APUC)  make  up  less  than  10  percent  of  the  observed  breaches  in 
this  portfolio  over  the  time  observed.  These  results  are  not  totally  unexpected,  as  in  a 
more  mature  portfolio  we  would  expect  to  see  more  procurement  and  O&S  breaches 
and  fewer  RDT&E  breaches.  Over  half  of  the  schedule  breaches  and  nearly  three- 
quarters  of  the  O&S  breaches  are  attributable  to  three  programs  that  are  being  pro¬ 
cured  through  a  single  manufacturer  (MH-60S,  CH-53K,  and  UH-60M).  Some  O&S 
breaches  appear  to  be  due  to  changes  in  platform  service  life.  The  service  lives  for 
MH-60S  and  UH-60M  were  extended  from  20  to  25  years,  and  this  may  have  driven 
the  O&S  breaches  these  programs  experienced.  Schedule  breaches  for  the  CH-53K 
were  due  to  component  test  delays  (late  parts  deliveries)  and  qualification  test  failures. 


Figure  10.5 

Types  of  APB  Breaches,  Helicopter  Portfolio,  2002  Through  2012 


APUC  breach 
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O&S  breach 


Procurement  breach 
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14  For  specific  definitions  of  APB  breach  types,  see  Chapter  Nine  or  William  Parker,  Defense  Acquisition  Uni¬ 
versity,  Program  Manager’s  Toolkit,  January  2011. 
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These  same  program  delays  and  testing  issues  may  have  also  contributed  to  the  O&S 
breaches  experienced  by  CH-53K.  Some  of  the  schedule  breaches  for  the  MH-60S  can 
be  attributed  to  labor  actions  and  strikes  in  2006.  There  were  no  MILCON  or  O&M 
breaches  reported  for  this  portfolio;  however,  none  of  the  programs  had  associated 
acquisition  O&M  funding  (although  they  did  have  other  types  of  O&S  funding),15 
and  only  three  programs  in  the  portfolio  had  associated  MILCON  funding.  Two  of 
those  programs  (Comanche  and  VH-71)  were  cancelled  before  entering  Milestone  C. 
The  absence  of  MILCON  and  O&S  funding  for  most  programs  may  reflect  some  of 
the  unique  characteristics  of  this  portfolio;  i.e.,  most  programs  are  upgrades  and  modi¬ 
fications  of  existing  platforms,  thus  much  of  the  military  infrastructure  to  support 
these  platforms  is  likely  already  in  place.16 

In  terms  of  Nunn-McCurdy  breaches,  several  programs  that  incurred  significant 
and  critical  breaches  (see  Table  10.3).  The  Comanche,  ARH,  and  VH-71  programs 
(all  three  Force  Application  programs)  were  cancelled  shortly  after  reporting  a  Nunn- 
McCurdy  breach.  For  the  H-l  Upgrades  program,  the  unit  cost  growth  is  primar¬ 
ily  a  result  of  increased  prime  contractor  labor  and  material  cost,  Government  Fur¬ 
nished  Equipment  costs,  additional  Non-Recurring  Engineering  costs,  and  correction 
of  deficiencies  from  Operational  Evaluation  Phase  1 .  The  unit  cost  increase  also  takes 
into  account  additional  costs  resulting  from  the  Marine  Corps  expanding  the  H-l 


Table  10.3 

Nunn-McCurdy  Breaches,  Helicopter  Portfolio,  2002  Through  2012 


Program 

Type  of  Nunn-McCurdy  Breach 

Year 

Comanche 

PAUC  critical  (current  baseline) 

2002 

MH-60S 

PAUC  significant  (current  baseline) 

2005-2012 

ARH 

APUC  and  PAUC  significant  (current  baseline) 

2007 

H-1  Upgrades 

APUC  and  PAUC  significant  (current  baseline) 

2008 

VH-71 

APUC  and  PAUC  critical  (current  baseline) 

2009 

Longbow 

APUC  and  PAUC  critical  (current  baseline) 

2009 

NOTES:  The  breach  status  for  H-1  Upgrades  reflected  in  the  table  is  based  on 
the  prior  APB  dated  July  2007,  which  was  the  current  APB  when  Congress  was 
notified  of  the  significant  Nunn-McCurdy  breach  to  the  APUC  in  December 
2008.  The  prior  APB  of  July  2007  was  revised  on  December  22,  2008,  to  reflect 
approval  of  the  program's  Milestone  III  decision.  Subsequently,  changes  in  the 
procurement  phasing  of  UH-1Y  and  AH-1Z  remanufacture,  and  AH-1Z  new  build, 
increased  the  PAUC  current  estimate  by  approximately  1  percent,  but  this  does 
not  constitute  a  breach  compared  to  the  current  approved  APB  of  December 
2008. 


15  O&M  costs  are  a  type  of  O&S  costs  incurred  for  using  and  supporting  the  system,  such  as  personnel,  main¬ 
tenance  (unit  and  depot),  spares,  and  training.  MILCON  costs  are  associated  with  construction. 

See  Table  9.5  for  a  full  account  of  breaches  by  program. 


16 
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Upgrades  procurement  objective  by  69  aircraft  (23  UH-1Y  and  46  AH-1Z)  to  a  total 
of  349  aircraft.  This  change  increased  not  only  the  overall  procurement  cost  of  the  pro¬ 
gram  but  also  the  average  unit  cost,  since  the  additional  AH-1Z  aircraft  are  in  excess  of 
the  existing  inventory  of  AH-1W  airframes.  As  a  result,  the  additional  AH-lZ  aircraft 
will  need  to  be  built  new  without  the  cost  benefit  of  remanufacturing.  The  Longbow 
breach  in  2009  was  addressed  by  splitting  the  program  into  two  pieces. 

Investigating  Trends  in  Multimetric  Visualizations 

With  multimetric  visualizations,  it  may  be  necessary  to  examine  the  relationships 
between  the  various  metrics  and  uncover  potential  outside  factors  that  may  have  con¬ 
tributed  to  trends  or  patterns  over  time.  In  our  case,  the  changes  in  shape  of  our  “radar 
chart  assessments”  shown  in  Figure  10.6  inspired  several  questions,  outlined  above,  on 
which  a  policymaker  may  require  additional  information. 

The  larger  shape  in  2004  in  Figure  10.6  appears  to  have  resulted  because  the 
number  of  programs  in  the  portfolio  dropped  to  six,  and  therefore  the  issues  encoun¬ 
tered  by  those  six  programs  became  more  heavily  influential  within  the  overall  portfo¬ 
lio.  By  2005,  three  new  programs  began  reporting  SAR  data,  which  led  to  the  signifi¬ 
cantly  changed  shape  from  the  previous  year  because  newly  reporting  programs  were 
typically  at  or  near  the  baseline  for  cost  metrics  and  would  not  have  recorded  quantity 
changes  from  the  previous  year.  Between  2006  and  2008,  the  AR1T  was  a  big  driver 
for  the  increase  in  the  size  of  the  shape  in  the  radar  chart,  with  five  breaches  alone 
recorded  for  this  program  in  2007.  Program  cancellations  of  the  AR1T  and  the  VH-71 
improved  the  status  of  the  overall  portfolio  in  2008  and  2009  as  did  the  rebaselining 
of  several  ongoing  programs  within  the  portfolio.  By  2012,  the  portfolio  consisted  of 
more  mature  and  stable  programs  that  reached  production  and  Milestone  C,  which 
contributes  to  the  smaller  shape  of  the  corresponding  radar  chart. 

Potential  for  Future  Cost  Growth  in  the  Portfolio 

Positive  trends  in  the  helicopter  portfolio  appear  to  be  driven  by  program-specific  events 
and  evolution  rather  than  by  improvements  in  management,  execution,  oversight,  or 
systematic  changes  in  underlying  factors  likely  to  affect  program  performance.  As  a 
result,  it  is  difficult  to  predict  whether  the  portfolio’s  recent  improvement  in  unit  cost 
growth  will  continue  or  reverse.  Although  the  CFI-47F  had  the  highest  cost  growth  of 
any  of  the  active  programs  in  the  2012  portfolio,  the  program  appears  to  be  executing 
well,  excluding  events  outside  the  program’s  control.  As  of  2014,  it  was  in  full-rate  pro¬ 
duction  with  multiple  FMS  customers  and  was  awarded  a  Block  II  contract  in  March 
2014.  It  is  likely  for  this  program  that  cost  growth  will  remain  stable  unless  affected  by 
external  events.  If  there  is  one  program  to  look  for  unit  cost  growth  in  the  next  year, 
it  might  be  the  M1T-60R  because  of  proposed  quantity  changes — particularly  if  the 
quantity  changes  are  not  associated  with  adjustments  to  the  APB.  Another  program 
that  may  need  to  be  watched  for  potential  cost  growth  is  the  C1T-53K,  which  entered 
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Figure  10.6 

Aggregate  Portfolio  Performance,  Helicopter  Portfolio,  2003  Through  2012 
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flight  testing  in  2014  and  has  already  incurred  with  some  delays  because  of  techni¬ 
cal  issues  that  will  require  some  gear  box  redesign.17  In  addition,  as  noted  above,  the 
CH-53K  includes  significant  changes  from  its  predecessor,  and  this  new  development 
may  bring  with  it  new  risks  and  potential  costs.  Furthermore,  the  program  narrowly 
avoided  a  breach  in  2009  because  it  increased  the  procurement  quantity  from  156  to 
200  because  of  the  increasing  size  of  the  Marine  Corps  authorized  end  strength.  How¬ 
ever,  since  the  Marine  Corps  will  be  shrinking  its  force  structure,  the  CH-53K  may 
experience  a  breach  when  the  production  quantities  are  finally  reduced  to  match  the 
future  smaller  size  of  the  Marine  Corps. 

Finally,  new  programs  may  enter  the  portfolio  and  affect  the  portfolio’s  unit  cost- 
growth  performance. 

Helicopter  programs  that  are  currently  in  predevelopment  stages  may  be  sources 
of  future  portfolio  cost  growth.  The  VXX  development  program  that  will  enter  the 
helicopter  portfolio  in  the  near  future  is  intended  to  provide  the  presidential  helicop¬ 
ter  with  the  capability  to  replace  the  cancelled  VH-71  program.  Given  the  previous 
political  scrutiny  on  the  VH-71,  the  VXX  may  also  be  a  high-visibility  program.  The 
program  is  a  modified  military-off-the-shelf  variant  of  Sikorski’s  H-92  Superhawk  mil¬ 
itary  transport.  Although  Canada’s  procurement  of  the  aircraft  has  suffered  from  some 
delays  and  missed  milestones,  the  maturity  of  the  platform  could  potentially  limit 
some  technical  risk  in  the  VXX  program.  In  addition,  the  EMD  contract  is  an  FPIF- 
type  contract,  which  will  limit  some  of  the  government’s  liability  for  cost  growth. 
Another  program  in  the  pre-MDAP  phase  that  could  potentially  contribute  to  portfo¬ 
lio  cost  risk  is  the  AAS  helicopter.  This  potential  program  is  another  attempt  to  replace 
the  Army’s  Kiowa  Warrior  after  the  Comanche  program  was  dropped  in  2003. 


17  “First  Flight  of  CH-53K  Heavy-Lift  Helicopter  Delayed,”  InsideDefense,  July  17,  2014. 


CHAPTER  ELEVEN 


Summary  and  Way  Ahead 


Part  Two  of  this  report  develops  a  methodology  and  set  of  metrics  that  can  be  used 
to  characterize  the  status  and  risk  of  portfolios1  of  MDAP  programs  over  time  and 
across  commodity  types.  As  noted  in  the  introduction,  being  able  to  easily  summarize 
the  status  and  performance  of  a  set  of  acquisition  programs  is  valuable  to  the  defense 
acquisition  community,  because  it  allows  analysts  to  understand  and  identify  sources 
of  risk,  areas  of  vulnerability,  lines  of  development  where  investment  is  needed,  and 
areas  of  strength  and  strong  program  performance. 


Summary 

The  report  defines,  explains,  and  demonstrates  a  portfolio  analysis  framework,  which 
included  the  following  steps:  identify  objectives,  choose  a  portfolio  type,  select  data 
and  metrics,  address  data  anomalies  and  challenges,  calculate  metrics,  and  visualize 
metrics.  We  provide  a  number  of  different  data  sources  that  could  be  used  for  analysis 
and  ways  to  overcome  some  of  the  challenges  associated  with  these  data  sources.  The 
methodology  also  includes  a  large  number  of  portfolio  metrics  and  visualizations  of 
those  metrics  that  can  be  used  to  summarize  the  findings  of  a  portfolio  assessment.  In 
selecting  metrics,  we  chose  a  set  of  metrics  that  would  comprehensively  assess  program 
and  portfolio  status  and  that  captured  many  dimensions  of  program  and  portfolio  risk 
and  status.  Having  many  different  metrics  is  important  to  the  portfolio  assessment 
framework  because,  as  we  noted  throughout,  each  metric  must  be  interpreted  in  the 
context  of  others  and  no  single  metric  provides  a  sufficient  basis  on  which  to  evaluate 
a  portfolio’s  performance.  The  visualizations  we  provide  are  also  valuable,  because  they 
can  help  analysts  interpret  a  given  metric  or  set  of  metrics,  particularly  when  the  focus 
of  the  analysis  is  change  in  performance  over  time. 


1  Subportfolios  include  smaller  sets  of  MDAPs  with  similar  characteristics.  The  analysis  of  a  subportfolio  allows 
us  to  study  the  status  of  these  smaller  sets  of  programs  in  more  detail  and  to  separate  their  status  from  the  statuses 
of  the  overall  portfolio  of  MDAPs  and  other  subportfolios. 
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In  addition  to  defining  a  methodology,  we  use  the  report  to  demonstrate  and 
validate  our  approach  and  selection  of  metrics  using  several  sample  portfolios,  chosen 
specifically  to  provide  a  robust  test  of  the  methodology  and  to  allow  for  refinements 
as  necessary.  Our  demonstration  and  application  to  the  helicopter  and  satellite  port¬ 
folios  included  descriptors  such  as  percentage  of  programs  at  Milestones  B  and  C  and 
number  and  type  of  breaches,  and  indicators  to  include  unit  cost,  percentage  of  funds 
remaining,  and  percentage  of  programs  with  any  quantity  change.  We  also  looked  at 
multimetric  charts  that  provide  a  more  aggregate  perspective  on  the  performance  of 
the  two  portfolios.  By  discussing  the  two  portfolios  side  by  side,  we  were  able  to  high¬ 
light  similarities  and  differences  between  the  two  portfolios  and  to  further  highlight 
the  value  of  specific  metrics  within  the  portfolio  assessment  process.  Throughout  the 
report,  we  focused  largely  on  defining  the  methodology  and  discussing  the  metrics 
and  did  not  fully  complete  the  assessment  by  digging  deeply  into  each  program  within 
the  portfolio.  In  Chapter  Ten,  however,  we  did  conduct  a  more  detailed  assessment 
of  the  helicopter  portfolio,  using  additional  graphs  and  our  metrics  to  offer  a  more 
robust  analysis  of  the  status  of  this  portfolio  of  programs.  That  chapter  also  provided 
some  insight  into  questions  raised  in  our  initial  discussion  of  metrics  in  Chapter  Nine, 
including  additional  detail  on  the  drivers  of  increases  and  decreases  in  unit  cost,  pro¬ 
grams  with  APB  and  Nunn-McCurdy  breaches,  and  the  rates  of  spending  of  programs 
within  the  portfolio.  This  expanded  narrative  is  a  small  example  of  how  the  methodol¬ 
ogy  provided  in  this  report  could  be  used  by  members  of  the  defense  acquisition  com¬ 
munity  to  study  and  assess  a  set  of  acquisition  programs  and  then  graphically  summa¬ 
rize  the  portfolio’s  overall  status. 


Way  Ahead 

This  report  provides  a  generalizable  and  repeatable  portfolio  assessment  methodology 
that  addresses  the  need  of  the  defense  acquisition  community  for  a  way  to  analyze  and 
communicate  the  status  of  a  set  of  acquisition  programs.  The  methodology  is  valuable 
because  it  can  be  applied  to  any  number  of  acquisition  portfolios  using  a  relatively 
straightforward  approach  and  because  we  have  included  possible  visualizations  which 
will  help  in  the  interpretation  of  key  metrics. 

The  work  presented  here  can  be  expanded  in  several  ways  to  make  the  method¬ 
ology  even  more  useful  for  analysts  and  to  allow  analysts  to  use  the  methodology  to 
inform  policymakers. 

First,  as  noted  throughout,  this  report  has  focused  on  the  development  and  vali¬ 
dation  of  the  methodology  for  portfolio  assessment  rather  than  on  the  complete  execu¬ 
tion  of  the  portfolio  assessment  itself.  One  next  step  would  be  to  apply  the  methodol¬ 
ogy  here  to  a  larger  number  of  portfolios,  focusing  more  explicitly  on  the  assessment 
phase  of  the  analysis  rather  than  so  heavily  on  the  methodology  and  the  metrics.  These 
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assessments  could  serve  several  purposes.  Most  important,  they  would  provide  insight 
into  the  status  of  a  larger  set  of  acquisition  programs  and  allow  analysts  to  provide  a 
more  comprehensive  picture  of  the  risk  and  performance  of  DoD’s  acquisition  portfo¬ 
lio  as  a  whole.  Conducting  a  larger  number  of  assessments  would  also  facilitate  cross¬ 
portfolio  comparisons,  which  would  allow  analysts  and  policymakers  to  identify  areas 
where  additional  investment  or  attention  is  necessary.  Finally,  the  process  of  conducting 
many  portfolio  assessments  using  the  methodology  defined  in  the  report  will  contrib¬ 
ute  to  the  refinement  and  improvement  of  the  methodology  itself  as  repeated  use  will 
highlight  areas  where  additional  metrics  or  alternative  visualizations  may  be  valuable. 

Another  way  that  the  work  here  could  be  expanded  builds  off  this  notion  of  refin¬ 
ing  and  improving  the  current  methodology.  Over  time,  the  objectives  and  interests 
of  policymakers  and  defense  acquisition  analysts  are  likely  to  change  and  evolve.  They 
may  become  interested  in  different  aspects  of  portfolio  performance  or  in  different  met¬ 
rics  as  ways  to  assess  this  performance.  The  methodology  in  this  report  can  be  easily 
refined  and  updated  to  reflect  these  new  priorities  and  interests  by  collecting  additional 
data,  adding  metrics,  and  developing  new  visualizations  that  can  address  emerging 
questions  and  concerns.  Even  in  the  near  term,  the  methodology  could  be  improved 
and  refined  by  including  additional  metrics  that  we  did  not  consider  in  detail  for  this 
report.  One  candidate  would  be  the  types  of  contracts  and  the  share  of  contracts  across 
various  contractors.  This  specific  metric  would  be  valuable  for  providing  insight  into 
the  status  of  the  industrial  base,  but  certainly  other  metrics  could  provide  insight  into 
other  aspects  of  portfolio  performance.  Refinement  of  the  methodology  should  not  be 
done  haphazardly,  however,  and  would  still  require  vetting  of  data  sources,  mitigation 
strategies  to  deal  with  data  anomalies,  and  careful  selection  of  visualizations  that  pack¬ 
age  the  metrics’  results  for  the  policymaker. 

Regardless  of  the  specific  direction  it  takes,  the  portfolio  assessment  methodology 
presented  in  this  report  should  have  wide  and  growing  applicability  within  the  defense 
acquisition  community  given  the  current  fiscal  environment.  It  provides  a  way  for 
analysts  and  policymakers  to  track  and  understand  portfolio  performance  and  to  iden¬ 
tify  and  address  problems  that  arise  sooner  rather  than  later. 


APPENDIX  A 

JPALS  Program  History 


This  appendix  provides  a  summary  of  key  events  and  milestones  for  the  JPALS  effort 
over  time.  It  supplements  what  has  already  been  provided  in  the  main  text. 

Table  A.1 

JPALS  Key  Events  and  Milestones 


Date 

Event 

1992-1995 

MNS  was  based  on  efforts  that  started  in  1992,  by  the  Assistant  Secretary  of  Defense 
(ASD)  (AT&L)  to  migrate  away  from  ground-based  navigational  aids  (NAVAIDS)  and 
toward  GPS-supported  systems.  This  movement  was  in  line  with  FAA  actions  to  end 
production  of  the  Microwave  Landing  System  (MLS)  ground  stations  and  start  R&D  on 
GPS-based  navigation  systems 

August  1995 

MNS  for  a  precision  approach  and  landing  capability  was  approved  by  the  JROC 

May  1996 

Approval  for  the  JPALS  effort  to  enter  Phase  0;  Air  Force  designated  as  lead  agent 

October  1997 

Initial  AoA  for  JPALS  was  completed 

March  1998 

JPALS  System  Threat  Assessment  Report  (STAR)  was  initially  completed 

September 

1998 

Milestone  1  DAB  review  in  FY  1998  was  not  convened  because  of  funding  shortfalls; 
additional  program  guidance  provided  by  the  Deputy  Linder  Secretary  of  Defense 
(DUSD)  (A&T)  directed  the  continuation  of  premilestone  activities  and  prototype 
demonstrations  to  prove  operational  capabilities  and  maturity  of  the  AoA's  most 
promising  alternatives 

FY  2002 

Goal  was  to  reach  a  Milestone  I/ll  in  FY  2002;  however,  because  of  continued  service 
funding  shortfalls  in  the  FY  2002  POM,  this  date  also  was  not  met 

June  2004 

USD  (AT&L)  directed  that  the  original  AoA  be  updated  to  reflect  the  effect  of 
technological  advances,  new  capabilities,  and  improvements  to  other  alternatives  that 
may  have  arisen  since  the  original  analysis,  and  that  applicable  adjustments  be  made  in 
the  cost  estimates 

September 

2005 

On  implementation  of  the  JCIDS  process,  the  Joint  Staff  directed  that  the  MNS  be 
converted  to  an  ICD,  which  was  approved 

November 

2005 

AoA  update  was  validated  by  AFROCC 

February  2006 

JPALS  STAR  was  approved 
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Table  A.1  (continued) 


Date 

Event 

March  2007 

JPALS  CDD  was  validated  or  approved  by  JROC  (only  Increments  1  and  2  were 
approved);  Navy  designated  as  lead  agent 

July  2007 

Acquisition  Strategy  (AS)  signed  by  OSD  (AT&L) 

2nd  Qtr.  FY 
2007 

Navy  awarded  three  contracts  from  a  Broad  Agency  Announcement  with  a  total 
value  of  $12  million;  these  contracts  were  intended  to  stimulate  the  competitive 
environment,  perform  additional  sea-based  system  trade  studies,  review  potential 
candidate  architectures  and  corresponding  technology  maturities,  and  assess  potential 
growth  path  strategies  to  future  capabilities 

June  2008 

Defense  Acquisition  Executive  (DAE)  conducted  DAB;  designated  JPALS  as  an  MDAP 
ACAT  ID;  approved  Milestone  B  for  Increment  1A 

July  2008 

ADM,  AS,  and  §2366a  of  Title  10  Milestone  B  Certification  was  approved  by  USD  (AT&L) 

July  2008 

JPALS  SDD  or  development  contract  was  competitively  awarded  to  Raytheon 
Corporation 

July  2008 

Following  contract  award,  a  GAO  bid  protest  against  the  contract  award  was  issued 
with  an  associated  stop  work  order 

September 

2008 

Bid  protest  was  withdrawn  and  a  contract  restart  letter  was  issued 

December 

2008 

DAE  approved  APB 

January  2009 

SRR-2 

April  2009 

IBR 

June  2009 

SFR 

December 

2009 

PDR 

January  2010 

Increment  2  CDD  was  signed 

May  2010 

Navy  chaired  Configuration  Steering  Board  (CSB)  as  a  part  of  a  Gate  6  Post  PDR  (result: 
no  changes  to  CDD  or  AS) 

December 

2010 

CDR 

2010 

Other  milestones  for  2010:  completion  of  the  aircraft  performance  requirements 
specification  and  delivery  of  the  first  EDM  system  to  the  contractor  system  integration 
lab 

July  2011 

Completed  early  testing  of  the  GPS  receivers  onboard  LHD-1,  which  mitigated  several 
program  risks  before  the  beginning  of  formal  developmental  test 

August  2011 

Navy-chaired  CSB  as  part  of  a  Gate  6  review 

September 

2011 

Both  actions  from  CSB  were  successfully  closed  with  the  ASN  (RDA) 
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Table  A.1  (continued) 


Date 

Event 

October  2011 

Program  office  received  EDM  2 

November 

2011 

Program  office  received  Avionics  Test  Kits  (AVTKs)  2-4  in  preparation  for  government 
testing 

December 

2011 

Contractor  also  delivered  EDM  3  to  the  contractor  system  integration  lab 

February 

2012 

JPALS  Inc.  1A  successfully  executed  a  checkout  flight  with  an  AVTK-equipped  King  Air 
test  aircraft  against  EDM  2  at  Naval  Air  Station  (NAS)  Patuxent  River 

May  2012 

JPALS  Inc.  1A  conducted  a  TRR 

June  2012 

Commenced  integrated  test 

June  2012 

Quarterly  exception  SAR  was  being  submitted  to  report  a  delay  in  the  current  estimate 
for  Milestone  C  of  six  months  from  May  2013  to  November  2013,  resulting  in  a  schedule 
breach  to  the  APB 

October  2012 

EDM  5  installation  on  CVN-77  was  completed 

December 

2012 

Sea  trials  commenced 

2012 

Raytheon  delivered  the  final  five  of  eight  EDMs  and  the  last  of  the  five  AVTKs 

January  2013 

A  PDR  capturing  schedule  and  procurement  cost  breaches  was  received  by  the  MDA 

June  2013 

Resources  and  Requirements  Review  Board  (R3B)  convened  to  review  the  Navy  Precision 
Approach  Landing  Capability  to  include  re-scope  to  a  single  increment 

SOURCES:  Department  of  Defense,  JPALS  Inc.  1  SARs,  December  2009  through  December  2012; 
Department  of  Defense,  2008b;  Longuemare,  1996;  Department  of  Defense,  2005. 
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Assessing  the  "Test  Case"  Portfolio 


In  this  appendix,  we  present  the  results  of  our  assessment  of  the  “test  case”  portfolio 
using  the  chosen  portfolio  metrics.  This  test  case  portfolio  was  used  as  our  original 
validation  of  our  portfolio  assessment  methodology.  We  later  used  the  methodology  to 
assess  the  helicopter  and  satellite  portfolios,  documented  in  Chapter  Three.  We  start 
with  descriptors — metrics  that  describe  the  composition  of  the  portfolio  and  its  key 
characteristics  and  then  move  on  to  indicators  that  provide  insight  into  the  portfolio’s 
performance  on  cost,  quantity,  and  other  dimensions.  The  metrics  include  a  mix  of 
static,  “point  in  time”  assessments  as  well  as  dynamic  measures  that  consider  perfor¬ 
mance  across  years.  It  is  important  to  note  that  in  this  chapter,  we  focus  on  a  discus¬ 
sion  of  the  metrics  and  their  application  to  the  test  case  portfolio.  We  do  not  provide  a 
complete  assessment  of  this  sample  portfolio  or  attempt  to  explain  fully  the  drivers  of 
the  observed  trends. 


Selection  of  Initial  Test  Case  Study  Portfolio 

To  test  the  metrics  we  had  identified  as  relevant  to  an  assessment  of  portfolio  per¬ 
formance,  we  needed  to  select  a  portfolio  of  programs  from  the  larger  portfolio  of 
MDAPs.  To  aid  in  efficient  data  collection,  we  decided  to  draw  on  previous  RAND 
research  for  OSD  that  had  developed  a  methodology  for  anticipating  Nunn-McCurdy 
breaches.1  This  methodology  was  intended  to  reduce  the  approximately  100  MDAPs  to 
a  more  manageable  “watch  list”  of  programs  that  should  be  monitored  more  closely  for 
possible  breaches.  Previous  RAND  work  used  this  methodology  with  2009  and  2010 
data  to  identify  a  watch  list  of  programs. 

We  decided  to  use  this  watch  list  portfolio  as  our  initial  test  portfolio  for  a  number 
of  reasons.  First,  there  was  already  a  large  volume  of  data  that  had  been  systematically 
collected  by  RAND  analysts  for  these  programs  in  2009  and  2010.  Second,  this  port¬ 
folio  included  a  variety  of  programs  that  are  representative  of  all  of  the  services  and  dif¬ 
ferent  commodity  types.  This  allowed  us  to  see  if  there  were  consistencies  in  how  data 


1  Arena  et  al.,  2014b. 
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were  reported  across  services  and  commodities  and  to  identify  data  issues  that  might 
affect  how  portfolio  metrics  are  reported.  Finally,  these  programs  could  be  considered 
high  risk  relative  to  other  MDAPs.  By  selecting  this  test  portfolio  as  the  one  in  which 
we  intended  to  observe  how  our  selected  metrics  reflected  this  portfolio  risk.  Another 
reason  that  considering  this  set  of  programs  as  a  portfolio  is  that  it  provides  insight 
into  the  overall  status  of  a  set  of  programs  that  are  of  interest  because  they  are  high 
risk,  thus  providing  analysts  with  additional  information  on  the  overall  status  of  a  set 
of  programs  that  might  cause  problems  in  the  future. 

For  the  test  case  portfolio,  we  selected  32  programs  from  this  watch  list  that  were 
reported  in  the  SARs  for  both  2010  and  2012.  Table  B.l  lists  these  32  programs.  For 
each  test  case  program,  we  collected  data  on  our  selected  program  metrics  for  2010 
and  2012.  We  selected  these  two  years  to  draw  on  data  already  collected  for  2010  and 
to  compare  them  with  the  most  current  year  of  reported  data  (from  the  2012  SARs). 
Although  two  years  is  not  enough  time  to  detect  or  diagnose  trends,  our  comparison 
allowed  us  to  assess  our  ability  to  collect  the  same  metrics  in  different  years  and  pro¬ 
vided  some  insight  into  the  suitability  of  our  methodology  for  use  in  portfolio  assess¬ 
ment  over  time. 


Table  B.l 

Initial  Test  Case  Programs 


DoD 

Air  Force 

Army 

Navy 

AMF  JTRS 

Advanced  EHF 

Army  IAMD 

AIM-9X 

Chem  Demil-ACWA 

AMRAAM  (AIM-90) 

Excalibur 

CH-53K  Program 

HMS  (JTRS) 

FAB-T  Increment  1 

JLENS 

DDG  100  Destroyer 

JSF  (F-35) 

Global  Hawk  (RQ-4A/B) 

Patriot  PAC-3 

E-2D  AHE 

JTN 

GPS  IMA 

H-1  Upgrades 

JASSM  (Baseline) 

LCS 

NAVSTAR  GPS— Satellite 
and  Control 

LHA  6 

Reaper 

LPD  17  Class 

SBIRS-High 

MH-60S 

WGS 

RMS 

Tomahawk  (R/UGM-109E) 

Virginia- class  (SSN  774) 

VTUAV 

SOURCE:  Arena  et  ai.,  2014b. 
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Descriptive  Metrics 

Our  first  set  of  metrics  included  descriptors  of  the  composition  of  the  portfolio.  Our 
test  case  had  32  total  programs,  although  some  of  the  figures  include  33,  as  we  ulti¬ 
mately  decided  to  treat  the  two  blocks  of  the  AIM-9X  program  separately.2  The  follow¬ 
ing  metrics  provide  a  picture  of  the  portfolio  in  terms  of  commodity  and  type,  matu¬ 
rity,  program  size,  and  quantity  churn. 

Portfolio  Composition  and  Latest  Milestone  Achieved 

Figure  B.l  shows  programs  broken  down  using  the  DAMIR  portfolio  categories. 
This  breakdown  is  useful  because  it  relies  on  a  categorization  that  most  members  of 
the  acquisition  community  are  already  familiar  with  and  which  links  directly  with 
DAMIR  as  an  information  system  with  potentially  relevant  data.  The  figure  shows  that 
the  force  application  category  contains  the  most  programs  followed  by  the  net  centric 
and  protection  categories.  Two  categories,  force  management  and  joint  training,  are 
omitted  because  they  contain  no  programs  in  our  test  case  portfolio. 

Figure  B.2  shows  the  distribution,  by  commodity  type.  Common  commodity 
types  include  satellites,  C3I,  missiles,  and  ships.  There  are  also  a  good  number  of  air¬ 
craft  programs,  when  all  types  of  aircraft  are  considered  together.  Both  commodity 
type  and  the  distribution  of  programs  across  DAMIR  portfolios  are  the  same  in  the 
two  years  we  considered  in  our  analysis.  This  may  not  be  true  of  all  portfolios,  however, 


Figure  B.l 

Programs,  by  DAMIR  Portfolio 
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2  We  treated  the  two  blocks  separately  because  they  have  different  PNOs. 
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Figure  B.2 

Programs,  by  Commodity  Type,  2010  and  2012  (Same 
Composition  Both  Years) 


RAND  Mgl  171/8-B.2 


n  =  32 


especially  when  longer  time  periods  (in  which  we  are  more  likely  to  see  programs  enter 
and  exit  a  portfolio)  are  considered. 

Descriptors  can  also  give  insight  into  the  maturity  of  the  portfolio  at  any  given 
point  in  time  as  well  as  over  time.  Figure  B.3  compares  two  pie  charts,  one  for  2010 
and  one  for  2012.  Each  pie  chart  characterizes  the  portfolio  by  illustrating  the  number 
of  programs  that  have  passed  Milestone  C  and  the  number  between  Milestones  B 
and  C.  Unlike  the  previous  two  descriptors,  these  distributions  do  not  stay  the  same 
over  time.  A  comparison  of  the  two  graphs  clearly  shows  that  more  programs  have 
passed  Milestone  C  in  2012  (unsurprisingly)  than  in  2010.  This  suggests  that  the  2012 
portfolio  is  more  mature  than  that  in  2010. 

Program  Size 

Another  metric  of  interest  for  policymakers  is  total  program  dollar  value,  which  is  the 
median  size  in  monetary  terms  of  programs  in  a  portfolio.  As  noted  above,  policymak¬ 
ers  may  be  more  concerned  about  breaches  and  such  problems  as  unit  cost  growth  or 
volatility  in  quantity  when  programs  are  large  and  involve  more  money.  As  we  consid¬ 
ered  total  program  value  for  the  test  case  portfolio,  we  faced  a  challenge  posed  by  a  sig¬ 
nificant  outlier,  the  F-35  program.  This  program  is  considerably  larger  than  all  others 
included  in  the  portfolio  and  so  significantly  skews  the  program  value  portfolio  metric. 
To  address  this,  we  calculated  the  total  program  value  metric  both  with  and  without 
the  F-35.  The  difference  is  reasonably  substantial  and  the  resulting  metrics  provide  us 
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Figure  B.3 

Programs,  by  Last  Milestone,  2010  and  2012 

2010  2012 


n  =  32 


n  =  32 
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with  two  slightly  different  types  of  information.  The  metrics  including  the  F-35  tell 
us  the  “true”  median,  range,  and  standard  deviation  of  total  program  values,  whereas 
metrics  that  exclude  this  outlier  provide  a  more  realistic  assessment  of  the  size  and  size 
distribution  of  programs  in  the  portfolio. 

Figure  B.4  shows  the  box  plot  for  total  program  value  excluding  the  F-35.  There 
is  a  statistically  significant  decline  in  the  total  program  value  between  2010  and  2012; 
however,  Cohen’s  D  tells  us  that  the  size  of  this  decline  is  effectively  zero  and  not  sub¬ 
stantively  meaningful.  This  suggests  that  for  the  most  part,  the  overall  program  value 
of  our  test  case  portfolio  remains  the  same  across  the  two  years.  Once  we  include  the 
F-35,  this  affects  the  median  value  and  standard  deviation  but,  notably,  not  our  con¬ 
clusion  that  there  is  effectively  no  change  in  program  value  across  the  two  years.  This 
is  not  all  that  surprising,  as  the  same  programs  are  included  in  each  year  or  sample. 

In  addition  to  excluding  the  F-35,  Figure  B.4  also  excludes  “outside  values,” 
defined  above  as  those  points  that  lie  more  than  1.5  times  the  interquartile  range  from 
the  rest  of  the  data.  Figure  B.5  includes  these  values  and  is  intended  to  illustrate  the 
effect  of  these  additional  outliers  on  the  graphical  representation  of  program  value. 
Although  excluding  these  outside  values  would  further  reduce  the  median  of  program 
value,  we  chose  to  leave  these  values  in,  erring  on  the  side  of  inclusion  rather  than  risk¬ 
ing  overmanipulation  of  the  data. 


FY  2013  total  program  value  ($  billions) 
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Figure  B.4 

Total  Program  Value,  Excluding  the  F-35,  2010  and  2012 


•  Excludes  JSF 

•  Differences  between 
the  two  years  are  small 
but  statistically 
significant 
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Figure  B.5 

Total  Program  Value,  with  Outside  Values,  2010  and  2012 
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Percentage  of  Funds  Remaining 

Program  maturity  is  another  indicator  of  portfolio  risk;  as  a  portfolio  matures, 
its  sources  of  risk  change.  When  a  portfolio  has  many  programs  that  have  not  yet 
reached  Milestone  B,  the  majority  of  the  risk  is  associated  with  program  development. 
When  a  portfolio  has  many  programs  that  are  past  Milestones  B  and  C,  the  major¬ 
ity  of  risk  is  associated  with  procurement.  Understanding  this  difference  can  help  an 
analyst  describe  the  risk  inherent  in  the  portfolio  and  provide  context  for  the  portfolio’s 
performance  on  other  dimensions.  Above,  we  considered  portfolio  maturity  using  the 
number  of  programs  past  Milestones  B  and  C.  Another  way  to  assess  program  matu¬ 
rity  is  to  consider  the  percentage  of  funds  remaining,  as  this  indicates  how  much  of  the 
program  budget  has  been  spent  (and  therefore  how  close  the  program  is  to  completion). 
Figures  B.6  and  B.7  offer  two  perspectives  on  this,  one  that  uses  RDT&E  funds  and 
one  that  uses  procurement  funds.  We  again  use  box-and-whisker  plots  to  visualize  the 
changes  between  these  two  portfolios  on  these  two  metrics  over  time. 

Figure  B.6  shows  the  percentage  of  RDT&E  funds  remaining  in  each  of  the  two 
years.  The  graph  suggests  a  small  decrease  in  funds  remaining  between  2010  and  2012, 
which  is  not  surprising  given  that  only  two  additional  years  have  passed.  This  change 
would  also  be  consistent  with  a  gradual  maturing  of  the  portfolio  over  time.  A  t-test 
confirms  that  this  difference  is  statistically  significant  (or  that  the  change  is  a  real 
change  in  the  trend  and  is  not  simply  noise  in  the  data);  Cohen’s  D,  however,  indicates 
that  the  effect  size  is  small.  This  graph  then  generally  confirms  our  sense  that  the  port¬ 
folio  in  2012  has  become  more  mature  but  also  suggests  that  this  change,  at  least  as 
measured  by  remaining  RDT&E  funds,  is  fairly  small. 

Figure  B.6 

Percentage  of  Funds  Remaining,  RDT&E,  2010  and  2012 


•  Differences  between 
years  are  statistically 
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Figure  B.7 

Percentage  of  Funds  Remaining,  Procurement,  2010  and  2012 
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Figure  B.7  shows  the  percentage  of  procurement  funds  remaining  in  each  of  the 
two  years.  It  also  suggests  a  decline  in  the  median  percentage  of  funds  remaining  (a 
sign  of  program  maturity)  but  also  a  sizeable  increase  in  the  distance  between  the  25th 
and  75th  percentiles  or  the  spread  of  the  data.  The  two  trends  together  suggest  that 
between  2010  and  2012,  some  programs  underwent  a  large  reduction  in  their  per¬ 
centage  of  procurement  funds  remaining  (in  other  words,  they  spent  a  large  amount 
of  their  procurement  budgets),  whereas  others  experienced  only  a  small  change.  A 
t-test  suggests  that  this  effect  is  statistically  significant  and  Cohen’s  D  tells  us  that  the 
effect  size  is  medium  and  substantively  meaningful.  Of  course,  it  is  also  worth  con¬ 
sidering  the  starting  points  for  the  two  metrics.  Although  most  programs  had  already 
spent  more  than  50  percent  of  their  RDT&E  budget  in  2010  (the  median,  in  fact, 
was  below  20  percent  remaining),  the  median  for  procurement  funds  remaining  was 
about  70  percent.  Therefore,  there  was  more  room  for  a  more  substantively  meaningful 
decline  when  considering  only  procurement  funds.  The  difference  in  starting  points 
makes  sense,  given  the  fact  that  RDT&E  funds  will  be  spent  early  in  a  program  life 
cycle,  but  procurement  funds  may  not  be  spent  until  later. 

Portfolio  Churn:  Quantity  Changes 

Turning  to  an  assessment  of  changes  in  quantity,  it  is  important  to  keep  in  mind  that 
a  change  in  quantity  does  not,  in  itself,  indicate  a  problem  in  the  program.  Rather,  it 
may  suggest  an  increase  in  demand  or  some  other  explainable  change.  As  an  example, 
the  Tactical  Tomahawk  experienced  a  sharp  increase  in  quantity  during  the  conflict 
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in  Libya  when  it  was  in  much  higher  demand.  This  did  not  suggest  a  true  problem  in 
the  program  but  instead  the  nature  of  U.S.  military  activity  and  requirements.  How¬ 
ever,  changes  in  quantity  demanded  or  procured  can  often  lead  to  instability  in  the 
program  that  increases  the  risk  for  cost  growth  or  other  problems.  Figure  B.8  uses  a 
bar  chart  to  highlight  programs  that  experienced  a  change  in  quantity  between  2010 
and  2012.  One  key  observation  is  that  not  all  programs  experience  changes  in  quantity. 
Our  churn  metric  measures  the  number  of  programs  with  any  changes  in  quantity  in 
a  given  year.  This  metric  provides  additional  insight:  18  out  of  the  33  programs  (again 
counting  the  AIM-9X  variants  as  two  separate  programs)  have  some  change  in  quan¬ 
tity  in  each  of  the  two  years,  although  it  is  not  the  same  programs  experiencing  quan¬ 
tity  changes  in  each  of  the  two  years.  Figure  B.8  identifies  programs  with  large  changes 
in  quantity.  On  the  positive  side,  SBIRS-High,  LHA  6,  and  JTRS  HMS  experience 
an  increase  in  2010,  whereas  PACO  and  LHA-6  experience  increases  in  2012.  Most 
of  the  negative  changes  also  occur  in  2012,  namely,  to  the  JLENS,  Global  Hawk,  and 
AMF  JTRS  programs. 


Performance  Metrics 

Nunn-McCurdy  and  Acquisition  Program  Baselines  Breaches 

Another  set  of  metrics  of  interest  to  policymakers  and  analysts  conducting  a  portfolio 
assessment  counts  the  number  of  both  Nunn-McCurdy  and  APB  breaches.  Figures 
B.9  and  B.10  show  the  number  of  each  type  of  breach,  comparing  2010  and  2012. 
Figure  B.9  shows  number  of  programs  with  significant  and  critical  breaches  in  the 
current  and  original  APUC  and  PAUC.  Figure  B.10  shows  the  number  of  new  APB 
breaches,  by  type,  in  each  year.  It  appears  that  there  are  somewhat  fewer  APB  and 
Nunn-McCurdy  breaches  in  2012  than  in  2010.  However,  when  we  assess  the  statisti¬ 
cal  significance  of  this  difference,  using  a  t-test  as  described  in  Chapter  Two,  we  find 
that  the  difference  is  not  statistically  significant.  This  means  that  the  change  in  the 
number  of  breaches  between  the  two  years  could  just  be  due  to  chance  and  does  not 
represent  a  meaningful  change  over  time.3 

Unit  Cost  Growth 

The  next  set  of  metrics  that  we  consider  focuses  on  unit  cost  growth — a  metric  of  sig¬ 
nificant  importance  and  interest  to  policymakers.  We  assess  unit  cost  growth  in  several 
ways,  first  looking  at  the  distribution  of  growth  in  APUC  and  PAUC  in  2010  and  2012 


3  The  t-test  in  this  case  compares  the  distribution  of  breaches  (numbers  and  types  of  breaches)  in  2010  and  2012 
to  determine  if  the  two  samples  appear  to  come  from  a  single  or  two  different  distributions.  Because  the  result  is 
not  statistically  significant,  we  cannot  reject  the  null  hypothesis  that  there  is  no  difference  between  the  two  years 
or  say  for  sure  that  the  small  change  we  do  observe  is  not  due  to  chance. 
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Figure  B.8 

Percentage  Change  in  Quantity  Versus  Baseline,  2010  and  2012 
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Figure  B.9 

Nunn-McCurdy  Breaches,  2010  and  2012 
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Figure  B.10 

Number  of  APB  Breaches,  2010  and  2012 


3 

3 

i/i 

E 

03 

s- 

o> 

o 


<u 

-Q 

E 

3 


Type  of  APB  breach 


NOTE:  Difference  in  number  and  types  of  breaches  between  2010  and  2012  is  not  statistically 
significant  at  p  =  0.2. 


RAND  MG1171/8-B.10 


148  Assessing  the  "Test  Case"  Portfolio 


across  all  programs  in  the  portfolio  and  then  comparing  the  cost  growth  in  2010  and 
2012  at  an  aggregated  portfolio  level.4 

Starting  with  the  unit  cost  growth  distribution  charts,  Figures  B.ll  and  B.12 
show  the  percentage  change  in  unit  cost  for  the  PAUC  and  the  APUC,  using  the 
current  baselines  in  2010  and  2012,  and  reveal  significant  variability  in  cost  growth 
across  programs.  Programs  with  especially  high  cost  growth  include  the  Global  Hawk, 
FAB-T,  JLENS,  VTUAV,  and  Virginia-class  submarine.  It  is  worth  noting  that  we 
have  excluded  data  for  the  JLENS  program  from  2012  for  PAUC  because  it  is  a  sig¬ 
nificant  outlier  with  very  high  cost  growth,  which  distorts  the  rest  of  the  graph.  Also, 
other  programs  appear  to  have  missing  data.  These  missing  observations  are  programs 
for  which  we  could  not  collect  cost  growth  data  in  the  years  of  our  analysis.  As  noted 
above,  there  may  be  several  reasons  for  missing  data  points,  including  rebaselining, 
program  completion,  program  splits  that  result  in  new  blocks  or  component  programs, 
or  the  nature  of  the  program  itself.  For  example,  Chem-Demil-ACWA  is  essentially  a 
service,  not  a  physical  product,  thus  precluding  the  use  of  metrics  derived  from  quanti¬ 
ties,  including  unit  cost  growth. 

A  visual  review  of  the  two  graphs  suggests  somewhat  lower  cost  growth  in  2012 
than  2010.  However,  this  difference  requires  additional  investigation  and  testing  before 
we  can  draw  any  conclusions  about  portfolio  status. 

The  next  step  in  our  analysis  of  unit  cost  growth  is  to  compare  the  percentage 
unit  cost  growth  in  the  PAUC  and  APUC  directly  for  each  of  the  two  years  in  our 
analysis  to  explore  whether  there  are  statistically  significant  differences  in  this  metric  at 
the  portfolio  level.  For  this,  we  use  box-and-whisker  plots,  the  interpretation  of  which 
was  described  in  Chapter  Three.  The  shaded  box  in  each  case  contains  75  percent  of 
the  data  and  the  line  within  each  box  the  median  value.  Looking  at  this  graph  for  the 
APUC  (Figure  B.13),  it  appears  that  median  cost  growth  declined  between  2010  and 
2012.  A  t-test  confirms  that  this  difference  is  statistically  significant  (resulting  from  a 
real  change  in  the  trend,  not  simply  noise  in  the  data),  although  our  Cohen’s  D  test 
suggests  that  the  effect  size  is  substantively  small.  Still,  the  graph  suggests  some  change 
in  unit  cost  growth  and  also  a  more  narrow  spread  or  variance  in  the  data  in  2012. 
However,  it  is  important  to  note  that  this  graph  does  not  tell  us  anything  about  the 
drivers  of  this  change,  whether  they  reflect  better  program  management  or  simply  a 
more  mature  portfolio  of  programs.  The  answers  to  these  questions  would  be  part  of  a 
comprehensive  portfolio  assessment  but  are  outside  the  scope  of  this  appendix. 


4  APUC  is  calculated  by  dividing  total  procurement  cost  by  the  number  of  articles  to  be  procured.  Total  pro¬ 
curement  cost  includes  flyaway,  rollaway,  sailaway  cost  (that  is,  recurring  and  nonrecurring  costs  associated  with 
production  of  the  item  such  as  hardware/software,  systems  engineering  [SE],  engineering  changes  and  warranties) 
plus  the  costs  of  procuring  technical  data  (TD),  training,  support  equipment,  and  initial  spares.  PAUC  is  calcu¬ 
lated  by  dividing  the  Program  Acquisition  Cost  by  the  Program  Acquisition  Quantity.  The  PAUC  and  APUC  are 
the  subject  of  the  Unit  Cost  Reports  (UCRs). 
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Figure  B.11 

Percentage  Cost  Growth  Against  Current  Baseline,  2010 
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Figure  B.12 

Percentage  Cost  Growth  Against  Current  Baseline,  2012 
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Figure  B.13 

APUC  percentage  Unit  Cost  Growth  Current  Baseline,  2010  and  2012 
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Moving  to  the  PAUC,  Figure  B.14  shows  the  percentage  change  in  PAUC  between 
2010  and  2012  using  the  current  baseline.  Although  the  graph  does  suggest  a  decline 
in  unit  cost  growth,  a  t-test  suggests  that  this  difference  is  not  statistically  significant. 
Therefore,  we  cannot  say  for  sure  that  the  change  that  we  observe  in  PAUC  is  not 
simply  due  to  chance  or  that  there  is  any  evidence  of  a  real  change  in  portfolio  status 
based  on  these  results.  If  we  were  interested  in  completing  a  full  assessment  of  the  two 
portfolios,  we  would  want  to  dig  further  into  why  we  find  a  statistically  significant 
change  only  for  the  APUC. 

In  addition  to  analyzing  percentage  change  in  unit  cost  using  unweighted  data, 
we  also  explored  whether  weighting  unit  cost  growth  by  program  size  affected  the 
result  of  our  analysis.  This  weighting  by  program  size  may  be  important  because  cost 
growth  in  large  programs  may  be  more  damaging  to  the  overall  portfolio  than  cost 
growth  in  smaller  programs.  Although  the  weighting  does  change  the  calculated 
median  and  mean  values,  it  does  not  affect  the  general  observations  above.  We  find 
substantively  small  changes  in  percentage  unit  cost  growth  that  are  statistically  signifi¬ 
cant  when  considering  the  APUC  and  not  statistically  significant  when  considering 
only  the  PAUC. 

Another  important  iteration  on  our  cost  growth  analysis  looked  at  cost  growth 
using  original  rather  than  current  baselines.  Looking  at  unit  cost  growth  using  current 
baselines  can  be  misleading  if  programs  have  recently  rebaselined,  as  this  may  make 
their  cost  growth  appear  artificially  low.  In  our  test  case  sample  of  programs,  this  is 
most  likely  to  be  problematic  for  programs  that  rebaseline  between  2010  and  2012, 
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Figure  B.14 

PAUC  percentage  Unit  Cost  Growth  Current  Baseline,  2010  and  2012 
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as  this  will  affect  their  apparent  cost  growth  in  2012  and  bias  our  assessments  of  how 
portfolio  metrics  have  changed  over  time.  Table  B.2  lists  the  programs  that  rebaselined 
between  2010  and  2012,  thus  affecting  our  assessment  of  unit  cost  growth  within  the 
test  case  portfolio  over  time,  along  with  a  reason  for  the  rebaseline.  The  table  shows 
that  the  rebaselining  issue  will  affect  a  large  number  of  our  programs  and  so  could  have 
significant  implications  for  our  overall  assessment  if  not  considered  more  carefully. 

To  address  this  possible  confounding  factor,  we  also  study  unit  cost  growth  for 
the  APUC  and  the  PAUC  using  the  original  baseline,  which  for  most  programs  is 
constant  over  the  entire  time  period.  A  few  programs  have  new  current  and  original 
baselines  between  2010  and  2012.5  Looking  at  original  baseline  cost  growth  will  not 
address  or  correct  for  the  effects  of  rebaselining  for  these  programs. 

Figures  B.15  and  B.16  show  the  percentage  unit  cost  growth  for  APUC  and 
PAUC  using  original  baselines.  The  graphs  look  similar  to  those  using  the  current 
baseline,  and  the  general  insights  offered  by  each  graph  are  largely,  but  not  entirely, 
the  same.  The  graph  for  percentage  change  in  APUC  suggests  a  decline  in  unit  cost 
growth  between  2010  and  2012  that  is  statistically  significant.  Calculating  Cohen’s  D 


5  An  original  APB  may  be  revised  only  in  the  event  of  a  critical  breach,  which  entails,  for  the  original  APB,  an 
increase  of  at  least  50  percent  over  the  original  APUC  or  PAUC.  However,  the  current  APB  may  be  changed  for 
the  following  reasons:  at  milestone,  full-rate  production,  and  full  deployment  decisions;  at  a  major  program  struc¬ 
ture  approved  and  fully  funded  by  the  MDA;  or  if  the  MDA  concludes  that  fact-of-life  program  changes  deem 
management  to  the  existing  baseline  impractical.  Programs  are  explicitly  prohibited  from  rebaselining  simply  to 
avoid  a  breach.  DoDI  5000.02,  January  2015. 
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Table  B.2 

Programs  Experiencing  a  Rebaseline  Between  2010  and  2012 


Program 

Current 

Baseline 

Reason  for  Rebaseline 

AEHF 

2012 

Established  SV5-6  as  a  subprogram 

Army  IAMD 

2012 

APB  reflects  increased  production  quantity  and  added  systems 

CH-53K 

2013 

O&S  breach,  quantity  changes,  and  beyond-threshold  increases  in  PLCCE 

E2-D  AHE 

2013 

New  APB  for  full  rate  production  (FRP) 

Excalibur 

2012 

Program  restructured  due  to  Nunn-McCurdy  breach 

JSF/F-35 

2012 

APB  reflects  development  program  restructure  in  2011 

LHA  6 

2012 

APB  updated  to  include  LHA  7 

LPD  17 

2011 

APB  change  to  reflect  LPD  26  and  LPD  27 

Reaper 

2012 

Milestone  C  approved,  new  production  APB 

RMS 

2012 

APB  updated  to  reflect  new  KPPs  from  the  CDD 

SBIRS 

2013 

Established  GEO  56  Block  Buy  program 

Tomahawk 

2011 

APB  reflects  increased  procurement  quantities  (Libya  replenishment) 

VTUAV 

2011 

Program  restructured  due  to  APUC  breach 

NOTE:  Programs  in  blue  have  a  new  original  and  current  baseline  in  the  indicated  year. 


Figure  B.15 

Percentage  Change  in  APUC,  Original  Baseline,  2010  and  2012 
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Figure  B.16 

Percentage  Change  in  PAUC,  Original  Baseline  2010  and  2012 
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suggests  that  this  change  is  substantively  meaningful  and  large,  unlike  the  small  effect 
observed  for  the  current  baseline  graph  above.  This  suggests  that  rebaselining  does  have 
an  effect  on  our  assessment  of  unit  cost  growth.  In  the  case  of  the  APUC,  focusing  on 
the  current  rather  than  the  original  baseline  may  underestimate  the  negative  change 
in  unit  cost  growth  between  2010  and  2012.  However,  in  the  case  of  the  PAUC,  even 
looking  at  the  change  using  the  original  baseline  does  not  suggest  a  significant  change 
in  unit  cost  growth  between  the  two  years.  Although  the  median  does  decline  slightly, 
this  difference  is  not  statistically  significant  and  so  could  just  be  due  to  chance  or  noise 
in  the  data. 


Summary 

In  our  analysis  of  the  test  case  portfolio,  we  focused  primarily  on  describing  and  docu¬ 
menting  a  set  of  metrics  and  visualizations  that  can  be  used  in  a  subsequent  assessment 
of  how  and  why  the  test  case  portfolio  has  changed  over  time.  Although  we  do  not 
conduct  a  comprehensive  analysis  in  this  chapter,  our  visualizations  appear  to  provide 
some  evidence  of  a  gradually  maturing  portfolio:  More  programs  have  passed  Mile¬ 
stone  C,  and  programs  appear  to  have  spent  more  of  their  RDT&E  and  procurement 
funds.  There  has  been  a  small  but  statistically  significant  decrease  in  unit  cost  growth 
when  APUC  is  considered  alone  and  an  apparent  drop  in  the  number  and  types  of 
breaches  between  the  two  years  (although  this  change  was  not  statistically  significant). 
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We  also  observed  some  decrease  in  total  funds  remaining,  particularly  for  procure¬ 
ment,  and  some  churn  in  quantity  in  both  years  of  the  analysis.  These  trends  would  be 
consistent  with  a  maturing  portfolio  and  may  suggest  some  improvement  over  the  two 
years  of  our  analysis.  However,  before  making  any  strong  statements  about  the  status 
of  the  test  case  portfolio  in  the  two  years  of  our  analysis  or  any  changes  over  time,  we 
would  need  to  conduct  a  more  in-depth  assessment  to  explore  why  we  see  the  changes 
that  we  do.  It  is  important  to  note  that  the  conclusions  and  inferences  that  we  can  draw 
from  our  analyses  are  limited  by  that  fact  that  our  current  assessment  compares  only 
two  years  of  data  and  a  relatively  small  number  of  programs. 
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The  authors  examine  the  cause  of  the  Joint  Precision 
Approach  and  Landing  System  (JPALS)  major 
defense  acquisition  program  Nunn-McCurdy  unit 
cost  breach  and  document  a  methodology  that 
can  assess  and  summarize  the  overall  performance  of  an 
acquisition  portfolio  at  a  point  in  time  and  over  several  years. 
In  January  2014,  the  Navy  informed  the  USD  (AT&L)  that 
both  the  average  procurement  unit  cost  and  the  program 
acquisition  unit  cost  for  the  JPALS  Inc.  1A  program  exceeded 
critical  thresholds  against  both  the  original  baseline  and 
the  current  baseline,  triggering  the  Nunn-McCurdy  process, 
which  is  statutorily  required  by  the  2009  Weapon  Systems 
Acquisition  Reform  Act  legislation.  The  team  used  official 
primary  source  documentation,  interviews,  and  trade 
literature  to  assess  and  document  the  reasons  for  the  critical 
cost  growth.  The  methodology  developed  to  assess  portfolio 
performance  included  identifying  objectives,  choosing  a 
portfolio  type,  selecting  data  and  metrics,  addressing  data 
anomalies,  and  calculating  and  visualizing  metrics.  The 
authors  applied  the  methodology  to  two  sample  portfolios — 
helicopter  and  satellite — from  2002  to  2012.  They  considered 
cost  and  schedule  performance  over  time,  reasons  for 
changes  in  the  portfolios'  composition  and  maturity,  the 
drivers  and  implications  of  rates  of  program  spending,  the 
percentage  of  funds  remaining,  the  potential  for  future  cost 
growth,  the  effects  of  rebaselining,  and  trends  in  associated 
Nunn-McCurdy  breaches. 
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